PaperlayerView exact revision on arXiv

Source-linked AI summary

ARFBench: Benchmarking Time Series Question Answering Ability for Software Incident Response

Stephan Xie, Ben Cohen, Mononito Goswami, Junhong Shen, Emaad Khwaja, Chenghao Liu, David Asker, Othmane Abou-Amal, Ameet Talwalkar

arXiv:2604.21199v2cs.LGcs.CV

TL;DR

It remains unclear whether foundation models can reliably answer the time-series questions engineers ask in software incident response. ARFBench tests this with expert-supported questions grounded in real incidents, and its results show frontier VLMs lead existing baselines while model–expert complementarity sets a higher oracle frontier.

  • Problem

    It remains unclear whether modern foundation models can reliably answer practical time-series questions that support incident-response decisions.

    Source: §1 · p. 1

  • Method

    ARFBench is a multimodal, multiple-choice benchmark built from real software incidents, with expert-supported questions organized into progressively harder anomaly-reasoning tiers.

    Source: §1 · p. 1

  • Results

    Frontier VLMs lead existing baselines, and a model–expert oracle reaches 82.8% F1 and 87.2% accuracy.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The results identify complementary model and expert strengths and present joint time-series and vision-language modeling as promising for observability-focused TSQA.

    Source: §1 · p. 2, §6 · p. 11

    ARFBench is limited to single-turn QA and excludes open-ended incident-response questions such as mitigation strategies and linking anomalies to incident patterns.

    Source: §5 · p. 11

Abstract

from arXiv · show

Time series question-answering (TSQA), in which we ask natural language questions to infer and reason about properties of time series, is a promising yet underexplored capability of foundation models. In this work, we present ARFBench, a TSQA benchmark that evaluates the understanding of multimodal foundation models (FMs) on time series anomalies prevalent in software incident data. ARFBench consists of 750 questions across 142 time series and 5.38M data points from 63 production incidents sourced exclusively from internal telemetry at Datadog. We evaluate leading proprietary and open-source LLMs, VLMs, and time series FMs and observe that frontier VLMs perform markedly better than existing baselines; the leading model (GPT-5) achieves a 62.7% accuracy and 51.9% F1. We next demonstrate the promise of specialized multimodal approaches. We develop a novel TSFM + VLM hybrid prototype which we post-train on a small set of synthetic and real data that yields comparable overall F1 and accuracy with frontier models. Lastly, we find models and human domain experts exhibit complementary strengths. We define a model-expert oracle, a best-of-2 oracle selector over model and expert answers, yielding 82.8% F1 and 87.2% accuracy and establishing a new superhuman frontier for future TSQA models. The benchmark is available at https://huggingface.co/datasets/Datadog/ARFBench.

1 Introduction

ARFBench tests whether foundation models can answer the time-series questions engineers use during incident response, using expert-supported questions grounded in real production incidents. The paper finds frontier VLMs lead existing baselines, a TSFM–VLM hybrid is promising, and model–expert complementarity yields a higher oracle frontier.

1 Introduction

  • ARFBench comprises multiple-choice questions from real software incidents, supported by expert annotations and incident context, with three tiers of increasing difficulty.The benchmark uses Datadog internal telemetry with sensitive data anonymized by an LLM and manual review.Source: §1 · p. 1
  • Its tiers progress from detecting anomalies to identifying their properties and reasoning about relationships between anomalies across time series.Question categories cover event classification, magnitude, trend, periodicity, correlation, and temporal relationships.Source: §1 · pp. 1–2
  • 62.7% accuracy and 51.9% macro F1: GPT-5 leads pre-existing models, exceeding naive baselines by at least 17 percentage points in both accuracy and F1.The results also indicate that VLMs tend to outperform LLMs.Source: §1 · p. 2
  • The authors report that their post-trained TSFM–VLM achieves accuracy and F1 comparable to frontier models, and hypothesize joint training is important for TSQA performance.Prior hybrid TSFM–LLM approaches had been compared only with LLMs or weaker VLM baselines.Source: §1 · p. 2
  • 82.8% F1 and 87.2% accuracy: a model–expert oracle combines complementary strengths and establishes a new superhuman frontier for future TSQA models.The authors report substantially different error profiles between leading models and human experts.Source: §1 · p. 2
  • The paper analyzes multimodal inputs, training stages, and model errors, positioning its results as a promising avenue for TSQA in observability and incident response.The benchmark and code are publicly released.Source: §1 · p. 2

2 Related Work

Prior TSQA work includes synthetic benchmarks and real-world datasets, but many existing real-world benchmarks lack complex multivariate or multi-series reasoning and contextual human evidence. ARFBench differs by combining incident-derived observability data with expert annotations and multiple anomaly-reasoning tasks.

Evaluating Multimodal Time Series Reasoning and QA

  • Many real-world TSQA benchmarks use public datasets but lack multivariate or multi-series reasoning complexity, and often rely on synthetic anomaly injection or lack contextual human annotations.TelecomTS and Time-RA are described as similar, but neither evaluates multi-series reasoning or uses expert annotations.Source: §2 · p. 3
  • ARFBench evaluates eight anomaly-related reasoning tasks over real software-incident observability data, enriched with context and human annotations.The supplied comparison describes its real-world, multivariate, context-enriched, expert-annotated, and multi-series reasoning dimensions.Source: §2 · p. 3

Time Series Anomaly Detection

  • Unlike classical anomaly-detection benchmarks that predict exact anomalous time ranges, ARFBench frames incident analysis as multiple-choice questions about higher-level anomaly reasoning.Its approach treats presence, start time, and end time as separate classification tasks and allows equally valid time ranges.Source: §2 · p. 3
  • ARFBench also asks about anomaly types and relationships across time series, including anomaly categorization and cross-metric correlation.The benchmark’s questions are illustrated with tier-specific examples and time-series inputs in image or tabular form.Source: §2 · p. 4, Figure 2 · p. 4

Modeling Methods in TSQA

  • Unlike methods that train time-series encoders from scratch, the paper jointly trains pretrained TSFMs with VLMs for multimodal TSQA.The authors position this as a novel approach to combining pretrained time-series and vision-language models.Source: §2 · p. 4

3 ARFBench

ARFBench evaluates time-series question answering about anomalies using questions built from real software incidents. Its design emphasizes escalating reasoning, contextual interpretation, and multivariate relationships, with incident timelines and automated question generation supporting construction and verification.

3.1 Problem Definition

  • ARFBench questions progress from anomaly presence to anomaly properties and then relationships across time series, including temporal alignment and lead–lag reasoning.The categories span binary detection, localization and characterization, and paired-series comparison.Source: §3.1 · pp. 4–5

3.2 Key Design Decisions

  • Multiple-choice answers address ambiguity in anomaly boundaries, while signal meaning, temporal context, and structured multivariate groupings shape anomaly interpretation.The benchmark groups variates by meaningful attributes such as services or datacenters; an individual variate may appear normal outside its group.Source: §3.2 · p. 5

3.3 Software Incident Timelines as Expert Annotations

  • Incident timelines record engineers’ investigation discussions and reasoning; ARFBench focuses on their metric visualizations for anomaly and cross-metric analysis.The timelines include embedded visualizations spanning logs, traces, and metrics.Source: §3.3 · p. 6

3.4 QA Generation Pipeline

  • The pipeline sanitizes query data into captions, generates templated questions and answer choices with an oracle VLM, then filters and manually verifies the pairs.The VLM uses time-series plots and incident discussions; manual review checks accuracy and sensitive-data leakage.Source: §3.4 · p. 6
  • ARFBench’s high variate counts and long time series make model input representation difficult.Question generation draws on incident discussion context, and the resulting questions undergo filtering and verification.Source: §3.4 · p. 6, Table 2 · p. 7

4 Experimental Results

Frontier VLMs lead on ARFBench, the Toto TSFM-VLM is competitive with frontier models, and an idealized model–expert oracle achieves higher scores than models or humans alone.

4.1 Evaluation Metrics

  • Accuracy measures overall correctness, while multiclass macro-F1 averages per-class F1 equally to reflect balance across answer classes.Raw choices are mapped to semantic answer classes so F1 can be compared across questions; macro-F1 penalizes frequent-choice predictors that exploit label imbalance.Source: §4.1 · p. 7

4.2 Baselines and Model Details

  • The evaluation included proprietary and open-source VLMs and LLMs, post-trained time series LLMs, random and frequent-choice baselines, and four researchers in a user study.The study sampled 188 questions for users, who first had to achieve at least 90% accuracy on a 16-question calibration set; the oracle analysis paired a domain expert with Toto-1.0-QA-Experimental.Source: §4.2 · pp. 7–8

4.3 Training Multimodal TSQA Models

  • ARFBench post-training combined synthetic anomaly examples and manually labeled real examples, using a Toto–Qwen3-VL hybrid with a three-stage SFT-then-RLVR pipeline.The synthetic set contained 12,000 examples; 207 manually labeled real examples were augmented to 395. The hybrid’s variate embedding MLP aggregates across time to avoid context length scaling with series length.Source: §4.3 · p. 8

4.4 Results

  • 62.7% accuracy and 51.9% F1 for GPT-5; Toto-1.0-QA-Experimental reached 63.9% accuracy and 48.9% F1, while Qwen3-VL 32B trailed GPT-5 by 9.9 pp accuracy and 6.8 pp F1.Toto had the highest accuracy; GPT-5 had the highest F1. Time series LLMs lagged behind similarly sized VLMs.Source: §4.4 · pp. 9–10
  • 87.2% accuracy and 82.8% F1 were achieved by the best-of-2 model–expert oracle, which assumes a perfect judge selects a correct answer when available.The model and experts had qualitatively different error profiles: experts made fine-grained perceptual and instruction-following errors, while models made domain-knowledge errors and sometimes inconsistent hallucinations.Source: §4.4 · p. 10

4.5 Ablations

  • 15.4 pp accuracy and 11 pp F1 gains followed Toto-1.0-QA-Experimental’s final RLVR stage, after its second SFT stage scored below the base Qwen3-VL 32B model.Without one or both SFT stages, early RL mean reward was around 2–4 times lower than after both stages.Source: §4.5 · p. 11
  • For both GPT-4o and GPT-5, removing all time-series data left performance between random and per-category frequent choice; GPT-5 gained 23.4 pp accuracy and 24.2 pp F1 with plots but no caption versus no time-series data.GPT-4o’s accuracy improved without captions, although its overall F1 remained below full input, so added context was not uniformly used.Source: Table 4 · p. 10, §4.5 · p. 11

5 Limitations and Future Work

ARFBench omits open-ended incident-response questions and remains limited to single-turn QA; understanding TSFM-VLM performance also remains future work.

5 Limitations and Future Work

  • ARFBench excludes open-ended questions about mitigation strategies and links between anomalies and incident patterns, which may require further labeled data.Examples include whether to scale resources based on an anomaly and whether it is directly linked to a deployment.Source: §5 · p. 11
  • The benchmark is limited to single-turn QA and is not yet comprehensive for agentic tasks or QA..Source: §5 · p. 11
  • Further work is needed to understand why TSFM-VLM models perform well on some tasks and how to improve them..Source: §5 · p. 11

6 Conclusion

The paper presents ARFBench for testing software-observability time-series reasoning and reports promise from joint time-series and vision-language modeling. It positions the open-source benchmark as a resource for work on time series and software incident response.

6 Conclusion

  • ARFBench evaluates reasoning capabilities grounded in the software observability domain..Source: §6 · p. 11
  • The paper concludes that joint time-series and vision-language modeling shows promise..Source: §6 · p. 11
  • The authors hope ARFBench’s open-source release will support community work on time series and software incident response..Source: §6 · p. 11

A Additional Details on Benchmark Construction Pipeline

ARFBench questions are generated from incident time series and reviewed by authors, with query-derived captions providing anonymized context. The appendix also connects time-series reasoning to incident response and describes domain categorization.

A Additional Details on Benchmark Construction Pipeline

  • Questions from 63 incidents were generated from extracted time series, filtered for overlapping time ranges, and reviewed by authors, who corrected putative answers when needed.The extraction was limited to under 2,500 non-empty variates per incident.Source: A Additional Details on Benchmark… · p. 17

A.1 Time Series Caption

  • Time-series captions summarize internal query strings, which are LLM-anonymized and used as question context; Figure 5 illustrates query components and a resulting description.The construction uses query strings from internal users and systems, not customer data.Source: §A.1 · p. 17, Figure 5 · p. 17

A.2 Software Incident Timelines In Context of Incident Response

  • Time-series questions can support triage, mitigation, and root-cause analysis, including checking whether anomalous behavior returns to normal after mitigation.Comparing time series may help identify which system component is the source of errors.Source: §A.2 · p. 17

A.3 Observability Time Series Subdomains

  • The appendix classifies time series into observability subdomains using their captions, including infrastructure metrics such as CPU usage and GPU utilization.Figure 6 shows distributions of question categories and time-series domains.Source: §A.3 · p. 18, §1 · p. 18, Figure 6 · p. 18

5. Security: metrics that monitor the general security of a system, such as vulnerability messages

The appendix details answer-choice filtering, negative examples for paired-series questions, and the definitions and semantic bins for anomaly reasoning tasks. It also describes model inference settings and input preprocessing.

A.4 Answer Choice Filtering

  • Answer choices are filtered to five per category, while semantic classes define consistent bins for multiclass F1 evaluation.The categories include anomaly presence, channel identification, start and end time, magnitude, categorization, correlation, and indicator order.Source: §A.4 · p. 18, B Details on Question Categories · p. 18, §B.2 · p. 20

A.5 Negative Tier III Augmentation

  • Tier III is augmented with uncorrelated pairs from different incidents to smooth a label distribution biased by within-incident pair selection.The authors note that within-incident pairs may be unrepresentative of real-world model comparisons.Source: §A.5 · p. 18

B Details on Question Categories

B.1 Question Definitions

  • Anomaly categorization distinguishes level shifts, transient spikes, changes in seasonality, variance, and trend; correlation and indicator tasks assess cross-series relationships and temporal order.Multiple anomalies may occur, but the largest or most timeline-relevant anomaly is generally treated as correct.Source: §B.1 · p. 20

B.2 Semantic Binning Categories

  • The appendix specifies task-specific semantic bins, including correlation classes that distinguish no anomalies, single-series anomalies, uncorrelated anomalies, and correlated anomalies.Indicator classes encode leading, lagging, uncorrelated, perfectly correlated, or absent anomalies.Source: §B.2 · p. 20

C Additional Details on LLM Settings for Inference

  • Inference uses provider APIs for closed-source models and local libraries for open-weight models, with model-specific temperature and reasoning-token settings.Answer choices are randomly shuffled to reduce option-order bias.Source: C Additional Details on LLM Settings… · p. 21
  • For ChatTS, series longer than 1,024 timestamps are subsampled, and inputs with many variates are reduced to the highest-mean channels; some other models use interval selection or truncation to fit context limits.GPT-5 and Qwen3 32B use a single fitting interval when possible, otherwise truncating the middle at tested ratios.Source: C Additional Details on LLM Settings… · p. 21

D.1 Data

The post-training data combines synthetic time series and manually cleaned real incident examples, with generated reasoning traces used as supervision. Synthetic series incorporate seasonality and drift before anomaly injection.

D.1 Data

  • 12,000 synthetic QA pairs were generated from varied-length Gaussian-noise series with seasonality and drift, then balanced anomaly categories and randomly placed anomaly injections.Series ranged from 240 to 50,000 timestamps, and included level shifts, spikes, variance or seasonality changes, and flatlines.Source: §D.1 · pp. 21–22, Figure 7 · p. 22
  • Real training data comprised 207 manually labeled examples from April 1–7, 2025 incidents, augmented to 395 examples without overlap with benchmark data.The data was generated using the ARFBench pipeline.Source: §D.1 · p. 22
  • Synthetic reasoning traces generated by prompting a VLM were used as supervision in both SFT stages for synthetic and real training sets.The prompts supplied the question and correct answer to the VLM.Source: §D.1 · p. 22

D.2 Training Frameworks and Hardware

The training appendix describes hardware and frameworks for model training and an RL objective using group-relative advantages from rewards. The reported Toto-1.0-QA-Experimental model has the top accuracy among tested models.

D.2 Training Frameworks and Hardware

  • VLM post-training uses veRL on eight NVIDIA H200 GPUs, while Toto models use TRL with eight NVIDIA B200 GPUs for the two Toto training runs.The VLM training and inference use FP16 precision.Source: §D.2 · p. 22

D.3 RL Formulation.

  • The RL objective optimizes multimodal models using group-relative advantages computed from rewards, with the advantage defined by the reward’s group-normalized value.The reward is a 0–1 function.Source: §D.3 · pp. 22–23
  • Toto-1.0-QA-Experimental achieves the top accuracy among models tested on ARFBench.Table 7 reports accuracy across question categories and models.Source: Table 7 · p. 24

E.1 Per Category Results

Per-category accuracy and F1 results are presented in Tables 7 and 8.

E.1 Per Category Results

  • Per-category accuracy and F1 results are reported in Tables 7 and 8.Source: §E.1 · p. 24

F Error Analysis and Case Studies

The authors sampled 50 model errors and categorized them as incorrect perception, limited context usage, or instruction-following errors.

F Error Analysis and Case Studies

F.1 Inter-Annotator Agreement between Users

Inter-annotator agreement was generally substantial for Cohen’s kappa and tentatively acceptable for Krippendorff’s alpha, except between non-domain experts.

F.1 Inter-Annotator Agreement between Users

  • Agreement was assessed across annotator pairs using Cohen’s kappa and Krippendorff’s alpha.Source: §F.1 · p. 24
  • Cohen’s kappa ranged from 0.6 to 0.8 across annotator pairs, while Krippendorff’s alpha was at least 0.667 except between non-domain experts.Source: §F.1 · p. 24

F.2 Qualitative Comparison of Model Errors with Domain Experts

This section discusses complementary error patterns in experts and models and describes benchmark-question filtering rules. The filtering criteria address whether snapshots, time ranges, and answer choices provide enough information, including for single- and paired-query formats and anomaly questions.

F.2 Qualitative Comparison of Model Errors with Domain Experts

  • Experts tend to err on fine-grained perception and some instruction-following tasks, while models tend to err through hallucination and failures to use domain knowledge.The comparison frames the error patterns as complementary rather than identical.Source: §F.2 · p. 25
  • Examples show models missing contextual or domain information, reversing cause and effect, and failing to perceive a changing trend in a time series.ChatTS fails to use extra textual context; GPT-4.1 lacks domain knowledge and perceptual ability; Gemini 3 Pro reverses cause and effect; Toto-1.0-Qwen3 32B misses a changing trend.Source: Figure 8 · p. 26, Figure 9 · p. 27, Figure 10 · p. 28
  • Expert examples include errors in identifying exact anomaly start or end timings and in understanding or following instructions.The figures contrast these expert errors with model performance on the illustrated questions.Source: Figure 13 · p. 31, Figure 14 · p. 32

2. Anomaly Identification

  • Anomaly-identification questions ask which channel contains an anomaly and are retained only when the full time series is visible in the snapshot.Source: §2 · p. 39

3. Anomaly Start

  • Anomaly-start questions require a visible full series and answer choices within the snapshot’s time range that match its time scale.Filter the question if all choices fall outside the snapshot’s range.Source: §3 · pp. 39–40

4. Anomaly End

  • Anomaly-end questions use the same visibility, time-range, and time-scale checks as anomaly-start questions.Filter the question if all answer choices fall outside the snapshot’s range.Source: §4 · p. 40

5. Anomaly Magnitude

  • Anomaly magnitude is the maximum ratio of anomaly values to counterfactual values, or maximum absolute deviation when the counterfactual is zero.Answer choices should include magnitudes reasonable for the data scale.Source: §5 · p. 40

6. Anomaly Categorization

  • Anomaly-categorization questions use six labels: Level Shift, Transient Spike, Change in Seasonality, Change in Trend, Change in Variance, and No Anomaly.The filtering criteria say not to filter out questions in this category.Source: §6 · p. 40

7. Anomaly Correlation

  • Anomaly-correlation questions are filtered when the two series have completely non-overlapping time ranges, but retained when their ranges overlap.The overlap is determined from snapshot x-axis labels and incident-timeline timestamps.Source: §7 · pp. 40–41

8. Anomaly Indicator

  • Anomaly-indicator questions ask whether one series’ anomaly leads or lags another, and are filtered when their time ranges do not overlap.Overlap can be assessed from snapshot x-axis labels and incident-timeline timestamps.Source: §8 · p. 41

G.3 Prompt for Filtering Benchmark Questions

  • Questions are filtered only for listed reasons, including insufficiently identifiable series, unreadable snapshots, or empty answer options.The prompt specifies that the full series should be visible for anomaly-presence questions and sets visibility and time-range checks for other categories.Source: §G.3 · p. 39, §8 · p. 41

Related papers

Loading 2604.21199v2…