PaperlayerView exact revision on arXiv

Source-linked AI summary

Do Time-Series QA Systems Read the Time Series? Evidence Use and Reasoning Reliability

Zhuomin Chen, Jingchao Ni, Xu Zheng, Janki Bhimani, Mo Sha, Wei Cheng, Dongsheng Luo

arXiv:2610.05686v1cs.AI

TL;DR

Correct answers alone do not show whether time-series QA systems use numerical evidence or reason reliably from it. The paper evaluates four systems with a common benchmark, evidence interventions, and rationale audits, finding that aggregate performance can conceal prediction changes, fallback behavior, and unsupported or invalid reasoning.

  • Problem

    Answer accuracy alone does not reveal whether retaining or changing the numerical series affects predictions, or whether rationale claims are grounded and their inferences valid.

    Source: §1 · pp. 1–2

  • Method

    The authors evaluate four systems using COMMON-TSQA, six numerical-evidence interventions with questions and targets fixed, and audits of visible rationales.

    Source: §1 · p. 2

  • Results

    Specialization improves native task performance, but similar task-level scores can conceal prediction turnover, some interventions prompt simple fallback behavior, and rationales can contain unsupported claims or invalid inferences.

    Source: §7 · p. 10

  • Takeaways & Limitations

    Predictive performance, evidence-responsive behavior, and rationale reliability are distinct evaluation dimensions.

    Source: §7 · p. 10

    Interpretation of training-related limitations is bounded by public designs and checkpoints; undocumented training signals are not assumed absent.

    Source: §6 · p. 9

Abstract

from arXiv · show

In recent years, time-series question answering (QA) systems have made significant progress. However, generating a correct answer does not show whether retaining the supplied numerical series improves task performance, nor whether the prediction is sensitive to changes in that input. While some systems provide rationales, answer accuracy also does not show whether their numerical claims are grounded in the supplied series or whether the stated inference is valid. In this work, we focus on evaluating four time-series QA systems: TimeOmni-1, ChatTS, TimeOmni-VL, and Time-MQA. First, for three systems with released evaluation data, we reproduce their reported results and compare the performance of the systems with their backbones. Then, we introduce a benchmark named COMMON-TSQA, which collects public evaluation datasets from existing time-series benchmarks and unifies their sample representation, task definitions, and answer schemas, while evaluating each system through its own interface under common evaluation criteria. The evaluation uses the original condition and six interventions while keeping the question and target fixed. Our analysis shows that aggregate performance alone can obscure how systems use numerical evidence. Similar task-level scores can arise despite substantial changes in individual predictions. Some interventions induce simple fallback behavior rather than preserved task ability. We also evaluate rationales for factual grounding, inference validity, and consistency with the final answer. We find that rationales often contain time-series claims unsupported by the input. Moreover, the rationale audit shows that agreement between a rationale and its final answer can coexist with incorrect numerical descriptions or invalid intermediate inferences.

1 INTRODUCTION

Correct answers and visible rationales alone do not establish whether a system uses the numerical series or reasons reliably from it. The paper evaluates four systems with a shared benchmark, evidence interventions, and rationale audits.

1 INTRODUCTION

  • Final-answer accuracy cannot distinguish reliance on temporal structure, coarse numerical cues, or non-series information, and a coherent rationale may misdescribe the series or reason invalidly.The authors ask how predictions respond when numerical evidence changes and whether rationale claims are supported by the series and conclusions follow from stated facts.Source: §1 · pp. 1–2
  • The evaluation separates native-task specialization gains, prediction responses to changed numerical evidence, and the reliability of visible reasoning.The intervention analysis distinguishes prediction turnover, fallback behavior, and cases where the transformed series implies a different answer; rationale audits examine grounding, inference validity, and answer consistency.Source: §1 · p. 2
  • COMMON-TSQA evaluates four specialized systems on shared samples and scoring criteria across six time-series QA tasks.The benchmark retains source-defined tasks and targets while standardizing representation and scoring.Source: §1 · p. 2

2 RELATED WORK

Related work spans specialized time-series language models, tests of dependence on numerical evidence, and evaluations of generated reasoning. These strands motivate assessing both evidence use and rationale reliability.

Time-series language models

  • Time-series language models use varied designs, including dedicated encoders, unified numerical-text frameworks, multi-task corpora, reinforcement learning, pattern-aware alignment, and vision-centric representations.Examples include ChatTS (Xie et al., 2025), Time-MQA (Kong et al., 2025), TimeOmni-1 (Guan et al., 2026a), PATRA (Lu et al., 2026), and TimeOmni-VL (Guan et al., 2026b).Source: §2 · pp. 2–3

Evidence dependence and behavioral evaluation

  • Prior evidence-dependence studies find that strong forecasting or QA performance does not by itself establish substantial use of every available input source.Tan et al. (2024) report forecasting performance often remains largely unchanged after removing pretrained language models or replacing them with simpler modules; Merrill et al. (2024) find limited benefit from series access in a harder two-series setting.Source: §2 · p. 3

Faithfulness and reliability of generated reasoning

  • Reasoning-faithfulness research distinguishes plausible explanations from accounts that reflect factors influencing predictions, and uses evidence checks or interventions to evaluate them.Jacovi & Goldberg (2020) distinguish faithfulness from plausibility; Lanham et al. (2023) perturb intermediate reasoning, while Turpin et al. (2023) show rationales can omit biasing-feature influence.Source: §2 · p. 3
  • Time-series explanation work includes TSQUERYBENCH, which evaluates numerical correctness in generated explanations without reference answers.Source: §2 · p. 3

3 COMMON-TSQA: A COMMON EVALUATION BENCHMARK

COMMON-TSQA addresses comparability challenges by combining public evaluation samples from existing benchmarks into a shared cross-system evaluation. It retains source tasks and targets while standardizing sample representation and scoring.

3 COMMON-TSQA: A COMMON EVALUATION BENCHMARK

  • COMMON-TSQA uses canonical samples from existing evaluation sets because systems otherwise face different tasks, datasets, or splits, making performance differences potentially test-set dependent.Its six tasks broaden task and domain coverage while reducing dependence on one benchmark’s annotation conventions or question formats.Source: §3 · p. 3
  • The benchmark draws on public sources including GIFT-Eval, SciTS, TIME, TSB-AD-U, TSAQA, CaTS-Bench, FactoryBench, SenTSR-Bench, TSRBench, and TSR-Suite-OOD.Source: §3 · pp. 3–4
  • Records retain source observations, task meaning, targets, and metadata while standardizing fields needed for cross-system evaluation.QA records preserve question meaning, option semantics, and target labels; records also include source split, task and domain metadata, and source grouping information.Source: §3 · p. 4

4 EVALUATION SETTINGS

Four systems are evaluated through their native interfaces, with numerical evidence altered under controlled conditions while other sample information is held fixed. Task-specific metrics score the resulting predictions.

4.1 SYSTEMS

  • Four systems receive the same canonical sample and task content, rendered through each system’s native input interface.Source: §4.1 · p. 4

4.2 EVIDENCE INTERVENTIONS

  • Each sample is tested originally and under six numerical-series interventions, while the question, choices, task context, and original target remain fixed.Predictions are evaluated against the Original target; transformations include shuffling values, flattening each series to its mean, and replacing values with Gaussian draws parameterized by observed mean and standard deviation.Source: §4.2 · pp. 4–5
  • The other interventions substitute a compatible series, remove the series while retaining other information, or replace values with zeros while preserving length and structure.Source: §4.2 · p. 4

4.3 METRICS

  • Forecasting is scored with MASE, anomaly detection with binary accuracy, pattern recognition, causal diagnosis, and decision intervention with accuracy, and cross-series comparison with relation accuracy.Lower MASE indicates better forecasting; relation accuracy requires the canonical predicted relation to match the target.Source: §4.3 · p. 5

5 EXPERIMENTS

Specialization improves native task performance in the available matched-backbone comparisons, but intervention results show that aggregate scores can mask prediction changes and fallback behavior. Rationale audits also find unsupported numerical claims and invalid inferences, even when rationales agree with final answers.

5.1 DOES SPECIALIZATION IMPROVE NATIVE TASK PERFORMANCE? (RQ1)

  • TimeOmni-1, ChatTS, and TimeOmni-VL consistently outperform their matched backbones in available native comparisons.Reproduction results are close to reported values on most native metrics; larger discrepancies occur in forecasting, particularly for TimeOmni-1 on the OOD split.Source: §5.1 · p. 5

5.2 HOW DO SYSTEMS RESPOND TO CHANGES IN NUMERICAL EVIDENCE? (RQ2)

  • ChatTS and TimeOmni-VL retain similar pattern-recognition accuracy under Answer-only despite changing many individual predictions.Correct-to-wrong and wrong-to-correct transitions largely offset each other on jointly valid samples, so similar aggregate scores can hide which samples are answered correctly.Source: §5.2 · p. 6, Table 2 · p. 6
  • Answer-only induces fixed-class fallback in anomaly detection: three systems label every sample normal, while ChatTS labels every sample anomalous.The residual accuracy depends on the frequency of the selected class, rather than discrimination between normal and anomalous samples.Source: §5.2 · p. 6
  • Flat moves nearly all paired forecasts toward the Original-history-mean reference, but error more often falls for TimeOmni-1 and Time-MQA and rises for TimeOmni-VL.The shift toward a simple level forecast is not a uniform improvement in prediction.Source: §5.2 · p. 7, Figure 4 · p. 7
  • For cross-series cases where an intervention changes the correct answer, correct updates are more frequent than stale answers for every system.Scoring against recomputed answers increases accuracy on the eligible subset, but stale answers and other errors remain.Source: §5.2 · p. 7, Figure 4 · p. 7

5.3 WHERE DOES THE EVIDENCE-TO-ANSWER REASONING PATH FAIL? (RQ3)

  • Only about half of extracted numerical claims are grounded for either TimeOmni-1 or TimeOmni-VL, with different task-level weaknesses.TimeOmni-1 has lower grounding rates in forecasting and cross-series comparison than anomaly detection, while TimeOmni-VL has its lowest rate in anomaly detection.Source: §5.3 · p. 8, Figure 5 · p. 8
  • Fewer than half of outputs pass inference validity for either system, while conclusion consistency is substantially higher, especially for TimeOmni-VL.A rationale can state correct amplitudes of 14 and 7.29 yet infer that the latter is larger, with its final answer following that mistaken comparison.Source: §5.3 · p. 8, Table 3 · p. 9

6 DISCUSSION

Specialization leaves distinct reliability gaps: stronger task performance does not ensure grounded reasoning or appropriate responses to changed numerical evidence. Across systems, valid answers, evidence responsiveness, and reliable rationales remain separate requirements.

Training-design interpretation

  • The analysis identifies limitations that remain after specialization but does not attribute errors to a particular training choice.Undocumented training signals are not assumed absent.Source: §6 · p. 9

TimeOmni-1

  • TimeOmni-1’s task and chain-of-thought supervision can improve final-answer performance without ensuring reliable numerical claims or inferences.Its public objectives emphasize task-level outcomes, leaving intermediate numerical verification for further study.Source: §6 · p. 9

ChatTS

  • ChatTS errors occur within task families represented in its public training data, and its responses to changed evidence include both appropriate updates and incorrect responses.The observations motivate examining how training connects numerical evidence to the required answer rather than treating broad task coverage as sufficient evidence of reliable reasoning.Source: §6 · p. 9

Time-MQA

  • On 327 target-changed cross-series pairs, Time-MQA shows more correct updates and fewer stale answers after adaptation, with the largest update increases under Flat and Zero.Pattern-recognition validity rises while paired-valid accuracy falls; in causal diagnosis, both improve, so aggregate gains reflect different changes across tasks.Source: Figure 6 · p. 9, §6 · p. 9

TimeOmni-VL

  • TimeOmni-VL’s rationale conclusion can match its final answer while its numerical grounding and inference remain unreliable.Conclusion consistency is a narrower success than a numerically supported, valid derivation.Source: §6 · p. 9

Cross-system takeaway

  • Valid answers, appropriate responses to changed evidence, and grounded, logically valid rationales are distinct requirements that specialization need not improve together.Source: §6 · p. 10

7 CONCLUSION

The paper evaluates four time-series QA systems through native reproduction, numerical-evidence interventions, and visible-rationale audits. Its findings show that predictive performance, evidence-responsive behavior, and rationale reliability should be evaluated separately.

7 CONCLUSION

  • Similar task-level scores can conceal substantial prediction turnover, intervention gains can accompany shifts toward simple forecasting references, and original-target scoring can penalize appropriate updates when evidence changes the answer.Rationale audits separately assess supported numerical claims, valid inference, and consistency with the final answer.Source: §7 · p. 10

A NATIVE EVALUATION AND REPRODUCTION

The paper reproduces reported results for three systems and compares specialized systems with matched backbones, then describes COMMON-TSQA’s sources and common representation. The supplied passages identify the scope of these comparisons but provide no result values.

A.1 SYSTEMS

  • The four systems use different backbones and specialization procedures, including supervised fine-tuning, chain-of-thought supervision, reinforcement learning, and LoRA adaptation.Time-MQA’s released Qwen variant is trained on TSQA; ChatTS uses synthetic time-series–text data, while TimeOmni-1 and TimeOmni-VL use chain-of-thought supervision.Source: §A.1 · p. 12

A.2 NATIVE COMPARISON

  • Reproduction evaluates TimeOmni-1 on TSR-Suite ID and OOD, ChatTS on Evaluation Sets A/B, and TimeOmni-VL on the TSR-Suite-OOD reasoning route.The section compares results with reported results and evaluates matched backbones on the same tasks; the table captions provide no numerical outcomes.Source: §A.2 · pp. 12–13, Table 5 · p. 13, Table 4 · p. 13

B.1 COMPOSITION AND SOURCES

  • COMMON-TSQA draws on public evaluation datasets and standardizes fields needed for a shared representation while retaining source-defined tasks and targets.Samples are adapted to each system’s interface, then outputs are converted to benchmark format and scored using task-specific criteria.Source: §B.1 · p. 13

B.2 TASKS

COMMON-TSQA groups source-defined questions into six time-series QA tasks and applies eligibility, conversion, and overlap checks. The intervention analyses examine answer transitions after evidence removal and forecasting shifts under Flat.

B.2 TASKS

  • The six tasks are forecasting, anomaly detection, pattern recognition, cross-series comparison, causal diagnosis, and decision intervention.Source: §B.2 · pp. 13–14

B.3 SOURCE SELECTION AND CONVERSION

  • Sources must preserve a mapping to one of the six tasks and provide provenance, split, target verification, output format, and scorer; candidate overlaps with available training data are removed.Input representations, option encodings, and output schemas are standardized while source task semantics and ground-truth answers are preserved.Source: §B.3 · p. 15

C INTERVENTION SEMANTICS

  • The interventions alter observed values in distinct ways: Shuffle permutes them, Flat replaces them with their mean, Noise draws Gaussian values, Swap substitutes a compatible series, Answer-only removes series input, and Zero replaces values with zero.Flat approximately preserves observed level while removing within-series variation; Zero preserves series length and channel structure but neither original level nor variation.Source: C INTERVENTION SEMANTICS · pp. 15–16

D.1 ORIGINAL–ANSWER-ONLY PREDICTION TRANSITIONS

  • Answer-only produces substantial answer changes in some settings with similar aggregate accuracy, but those transitions measure sensitivity to evidence removal rather than whether changed answers are appropriate.Source: §D.1 · p. 16, Table 9 · p. 17

D.2 FLAT AND SIMPLE FORECASTING REFERENCES

  • Movement toward the history-mean forecast is common across systems, while its effect on forecast error differs.The diagnostic compares Original and Flat on jointly scorable samples against a fixed history-mean reference; negative horizontal values indicate movement toward that reference, and negative vertical values indicate lower MASE under Flat.Source: §D.2 · p. 16, Figure 9 · p. 17

D.3 TARGET-AWARE EVALUATION OF CROSS-SERIES INTERVENTIONS

The target-aware evaluation separates responses to changed numerical evidence from responses where the correct answer remains unchanged. Its reporting distinguishes accuracy over all eligible pairs from update outcomes on target-changed pairs.

D.3 TARGET-AWARE EVALUATION OF CROSS-SERIES INTERVENTIONS

  • Intervention outputs are classified against both original and recomputed targets, separating correct updates, stale answers, and other outputs when the correct target changes.The categories do not require the original prediction to have been correct.Source: §D.3 · p. 17
  • The decomposition distinguishes errors on the transformed series from correct responses that no longer match the original answer.Source: §D.3 · p. 17
  • Table 10 reports original-target and recomputed accuracies over all eligible pairs, while update outcome shares use only target-changed pairs.Figure 4(b) normalizes correct updates and stale answers by their combined count; Table 11 separates interventions.Source: §D.3 · p. 17, Table 10 · p. 18, Table 11 · p. 18

E CASES

The cases illustrate that plausible numerical descriptions can still fail to answer the task or support a consistent conclusion. The accompanying adjudication prompt and release-scope discussion clarify how claims are checked and what comparisons the public artifacts permit.

E CASES

  • Series A’s amplitude is 14 and Series B’s is approximately 7.29, but the rationale reverses which series has the larger amplitude.Source: E CASES · p. 19
  • The pattern-recognition rationale mislocates the maximum and infers recurring-peak amplitude from one maximum, then gives a conclusion inconsistent with the final answer.The supplied values place the actual maximum at index 100, not the claimed index 72.Source: E CASES · p. 20

F LLM-AS-JUDGE PROMPT

  • On a blinded subset of 240 numerical claims and 100 rationale outputs across two systems and six tasks, human annotations showed strong agreement with LLM adjudication.Source: F LLM-AS-JUDGE PROMPT · p. 21, Table 12 · p. 21
  • The adjudication prompt extracts verifiable atomic series claims and labels them supported, contradicted, or not verifiable using the current example’s numerical evidence.It excludes answer correctness and system identity as evidence and requests concise numerical support for supported or contradicted claims.Source: F LLM-AS-JUDGE PROMPT · p. 21

G.1 PUBLIC TRAINING SIGNALS AND RELEASE SCOPE

  • Public releases lack complete intermediate training checkpoints; Time-MQA allows a fixed-base adapter contrast, but its original base revision is not pinned, so this is not an exact training-stage ablation.Source: §G.1 · p. 22

G.2 CHATTS TRAINING CORPUS

The ChatTS corpus audit finds candidate training signals but does not verify same-parent evidence-change supervision in the audited groups. A separate Time-MQA checkpoint contrast shows intervention- and task-specific changes rather than uniform adaptation effects.

G.2 CHATTS TRAINING CORPUS

  • ChatTS corpus tags are overlapping rule-based candidates, not gold annotations; the authors manually checked 57 stage-by-positive-category samples to assess tag semantics.Source: §G.2 · p. 22
  • The public generation pipeline includes numerical statements and source-series attributes, but the audit found no general deterministic per-claim numerical checker throughout the described pipeline.Source: §G.2 · p. 22
  • The audit verified no same-parent evidence-change and recomputed-target training pair among 50 candidate groups, a finding limited to those audited candidates rather than private training procedures.Source: §G.2 · p. 22

G.3 TIME-MQA BASE–LORA CHECKPOINT CONTRAST

  • Adaptation increases correct updates and reduces stale answers in the pooled comparison, with the largest correct-update increases under Flat and Zero rather than uniformly across interventions.The comparison covers 669 eligible sample–intervention pairs, including 327 whose correct target changes.Source: Table 15 · p. 23, Table 16 · p. 23, §G.3 · p. 23
  • The Time-MQA comparison holds the public base revision, tokenizer, input rendering, decoding, parsing, and scoring fixed; forecasting is excluded from interpretation because only two samples are scorable under both checkpoints.Source: §G.3 · p. 23
  • Validity and paired-valid accuracy change differently by task: pattern recognition gains validity but loses paired-valid accuracy, while causal diagnosis improves on both.Decision intervention raises validity without changing paired-valid accuracy, and cross-series comparison changes little.Source: §G.3 · p. 23, Table 17 · p. 24
Loading 2610.05686v1…