TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting Research

Motivation

• Only about one in five of 1,707 arXiv TSF papers since 2020 provides a code link, while libraries struggle to keep pace with new methods.TSFLab reports that TSLib grew from 18 to 41 methods despite over 1,200 new papers since 2024, and stopped maintenance in April 2026.Source: §1 · p. 1

Problem

• Agents using only their harness face stale TSF knowledge and inconsistent settings, which drive repeated searches, implementations, and experiments.The passage notes that an error in Informer’s official evaluation code influenced later study settings.Source: §1 · p. 2

Approach

• TSFLab combines domain guidance, structured knowledge cards, a unified experiment runner, and a Recursive Improver that verifies and integrates updated research resources.The workbench is updated independently of agent releases.Source: §1 · p. 2

Evaluation

• 20–163% higher task completion and 11–66% fewer fresh tokens per completed task are reported across baselines for reproduction and benchmarking.The paper also describes a pilot study of method development with fewer fresh tokens and less development time.Source: §1 · p. 3

TSF Libraries and Benchmarks

• TSF libraries and benchmarks provide implementations and evaluation pipelines, but manual maintenance leaves coverage lagging the literature.Results and rankings also vary with evaluation settings and tuning budgets, which agents must recover by reading library code.Source: §2 · p. 3

Research Agents and Harnesses

• TSF research agents either rely on general training knowledge or place domain support in an agent-specific harness that must be rebuilt for newer agents.TimeSeriesScientist, TimeCopilot, and MetaCaster produce forecasts rather than conduct TSF research [Zhao et al., 2025; Garza and Rosillo, 2025; Shen et al., 2026].Source: §2 · p. 3

Agent Workbenches

• General science and biomedical workbenches provide tools or procedures, whereas TSFLab targets TSF methods under declared evaluation protocols across agents and teams.Examples include ToolUniverse (Gao et al., 2025), Scientific Agent Skills (Kassis et al., 2026), and Biomni-E1.Source: §2 · p. 3

Self-Improving Agents

• Unlike systems that improve a single agent, TSFLab’s Recursive Improver checks methods against papers and official code, then merges verified resources and findings into the shared workbench.This lets research outputs from tasks contribute to workbench improvement and maintenance.Source: §2 · p. 3

3.1 Problem Statement

• The workbench leaves the agent’s model and harness unchanged while providing reusable domain knowledge, research procedures, and experimental tools.The stated goal is to improve valid task completion and reduce cost per completed task.Source: §3.1 · p. 3

3.2 TSF-Domain Guide

• Seventeen task-specific skills connect TSF conventions and pitfalls to workflows, checks, and procedures for tasks such as benchmarking new datasets.The example workflow identifies time fields, forecasting variables, sampling frequency, and evaluation settings, then checks partitioning, scaling, and metric consistency.Source: §3.2 · p. 4

3.3 Knowledge Structurer

• Knowledge cards expose summary, card, and source levels so agents can screen methods, datasets, and units, then retrieve deeper details as decisions require.Method cards connect mechanisms to use cases and configuration; 263 methods reuse at least one extracted shared unit.Source: §3.3 · pp. 4–5

3.4 Experiment Workflower

• The runner expands agent declarations into validated run matrices, applies shared data-splitting, training, and evaluation routines, and records results and experiment artifacts.It can resume failed runs from their last complete epoch and compare methods only under the same protocols.Source: §3.4 · p. 5

3.5 Recursive Improver

• Recursive Improver verifies incoming resources and outcomes, and accepted additions and fixes enter the shared workbench for subsequent research tasks.Checks cover usability, complexity, and consistency; accepted resources can include methods, units, datasets, and workflow repairs.Source: §3.5 · pp. 5–6

4.1 Experimental Setup

• The evaluation compares TSFLab with TSLib, TFB, and an empty-directory baseline across paper reproduction, benchmarking on ERCOT and Openmeteo, and a method-development pilot.The pilot searches on validation data and evaluates its final selection once on test data.Source: §4.1 · pp. 6–7

4.2 Main Results

• TSFLab completed all 12 reproduction attempts and 42/48 benchmark results; 73% of its 96 scheduled reproductions were within 10% of paper MSE, versus 21–46% elsewhere.The authors also report lower cost per completed outcome across reproduction and benchmarking.Source: §4.2 · p. 7
• Reproduction used fewer functions and 18–28% less added cyclomatic complexity than other settings, consistent with reuse of existing units; it still required new model code, while benchmarking reuses complete methods.Source: §4.2 · p. 7

4.3 Ablation Study

• Removing any workbench module lowered completion; without the guide, attempts used wrong ETTh1 split boundaries, while without the Workflower, agents introduced evaluation errors.Without the Knowledge Structurer, agents reused fewer units and added more code and complexity despite similar token use.Source: §4.3 · p. 8
• GPT-6 Sol and Sonnet 5.5 reduced tool calls by 48.5% and 51.1% versus GPT-6 Luna, while Luna and Sol completed all 12 attempts and Sonnet completed 11.Estimated costs for Sol and Sonnet were 18.2× and 9.0× Luna’s, respectively.Source: §4.3 · p. 8

4.4 Further Analysis

• TSFLab covers 44 of the 50 most cited methods, versus 29 across three other codebases; its median model implementation is 144 lines and its largest is 505.The cited figure tracks top-cited coverage, median and maximum model lines, functions per method, and intake tokens.Source: §4.4 · p. 8, Figure 4 · p. 8
• Extracting shared units reduced average functions per method from 11.9 to 8.6, and a refactor reduced maximum function complexity from 20 to 9 across 323 methods.The refactor preserved bit-identical parameters, outputs, and gradients on reference cases; 17 initial usability failures were repaired before integration.Source: §4.4 · pp. 8–9
• Model implementation accounts for 9% of method-intake tokens, while source reading, catalog search, and verification account for 28%, 27%, and 28%, respectively.Knowledge cards retain findings to help later agents locate methods and units without repeating the investigation.Source: §4.4 · p. 9

5 Conclusion

• TSFLab combines domain guidance, structured knowledge, reusable units, execution, and verified resource updates.It leaves the agent’s model and harness unchanged.Source: §5 · p. 9
• Across evaluated tasks, TSFLab improves task completion and token efficiency and supports method development.Ablations show complementary benefits from guidance, resource organization, and shared execution.Source: §5 · p. 9
• Retaining source knowledge, implementations, and experiment records lets new agents reuse verified results and repairs as tasks, models, and harnesses evolve.The authors frame this as support for cumulative TSF research across agents and teams.Source: §5 · p. 9

Limitations

• TSFLab’s effectiveness in other research domains remains untested because its guidance, resource descriptions, and experiment interfaces target TSF.Adapting it would require resources and checks suited to each domain’s tasks and conventions.Source: Limitations · p. 9
• Within the limited search time, method development has not produced a method that substantially outperforms all leading forecasting models.The reported improvements demonstrate support for the evaluated development tasks rather than a broadly superior forecasting method.Source: Limitations · p. 9
• More effective search strategies and experimentation over longer research cycles remain future directions.Source: Limitations · p. 9

Ethical Considerations

• TSFLab records sources, licenses, and reference revisions; reviews document paper-code discrepancies, and rankings are bounded by recorded protocols.Runs with network access should be sandboxed, with external actions requiring explicit authorization.Source: Ethical Considerations · p. 9

A.1 Framework Structure

• The agent selects resources, declares experiments, and interprets records, while four modules provide guidance, structured cards, shared execution, and resource checks.The flow proceeds from guidance and resource selection through configuration and execution to review and shared updates.Source: §A.1 · p. 13

A.2 Entry Instructions and Skills

• A harness-neutral entry file supplies conventions and verification requirements, while task-specific skills provide workflow steps and checks as needed.For new-data benchmarking, the procedures guide data-field and setting identification, registration, and inspection of splits and scaling.Source: §A.2 · pp. 13–14

B Knowledge Cards

• Knowledge cards separate structured fields, usage guidance, and source evidence, with automatic checks and read-only review validating fields and descriptions.Cards support summary, card, and source levels of access.Source: B Knowledge Cards · p. 14

B.1 Method Card: PatchTST

• PatchTST applies a channel-independent Transformer to overlapping patches, with a flatten head and RevIN, for long-horizon multivariate forecasting with long lookbacks.It is unsuitable when cross-channel interaction, exogenous inputs, probabilistic output, or calendar marks matter; patch_len must not exceed seq_len.Source: §B.1 · p. 14

B.2 Shared-Unit Card: RevIN

• RevIN normalizes each series using its window statistics and can apply an affine transform, then reverses the transform using cached statistics.It is intended for level or scale drift, but not when output channels differ from input or statistics must carry gradients.Source: §B.2 · p. 15

B.3 Dataset Card: ETTh1

• ETTh1 is represented by a dataset card (Listing 4).Source: §B.3 · p. 15

C.1 Declarative Configurations

• Experiments compose base, dataset, and method presets, then apply overrides and sweeps; data-dependent parameters can be resolved from the dataset at run time.Integrated methods can be declared without a separate training script, and the runner expands configurations into one run per sweep cell.Source: §C.1 · p. 16

C.2 Command Reference

• The command reference groups interfaces by workflow, with examples covering resource discovery, experiment execution and recovery, and research, comparison, and verification.The examples document workbench interfaces rather than a single shell script.Source: §C.2 · pp. 16–17

C.3 Budgets and Research Rounds

• An optional execution policy budgets runs, parallel jobs, wall time, and GPU-hours, while research rounds score the validation split unless overridden.The scheduler leases GPUs exclusively and counts all attempts against the budget.Source: §C.3 · p. 17

C.4 Records and Comparable Results

• Run records retain resolved configurations, metrics, checkpoints, and timing, while result reports rank methods only within a protocol and show seed coverage and spread.Records also carry protocol and model-variant fingerprints and feed the public leaderboard.Source: §C.4 · p. 17

D.1 Shared Setup

• Across settings, codebases were pinned and prompts, task files, and raw data were held constant; sessions used dedicated GPUs and fixed TSFLab snapshots.Sessions ran headlessly without persistent memory, plug-ins, MCP servers, or extra user instructions.Source: §D.1 · p. 18

D.2 Datasets

• Reproduction uses the first 14,400 ETTh1 rows across seven channels and Traffic, evaluating both datasets at horizons 96, 192, 336, and 720.ETTh1 is split chronologically into 8,640 training, 2,880 validation, and 2,880 test rows.Source: §D.2 · p. 18
• New-data benchmarking uses fixed ERCOT load and Openmeteo temperature exports with chronological splits, input 168, and horizon 24.ERCOT covers eight weather zones; Openmeteo covers 82 cities, and scaling is fitted on training data only.Source: §D.2 · p. 18

D.3 Paper Reproduction

• Paper reproduction selects recent arXiv methods absent from compared codebases that report per-horizon ETTh1 and Traffic MSE and MAE under the standard protocol.Eligible papers must provide enough training detail and train all four horizons in under one GPU-hour; each session has four hours and one GPU.Source: §D.3 · p. 18
• The four reproduction targets are RhyMix, StateFlow, AdaRDiff, and CoRe, with specified inputs and architectures but several unspecified training or architecture settings.RhyMix uses cyclic and multi-scale convolution paths; StateFlow uses recurrent hidden and residual-memory states; AdaRDiff wraps a linear backbone in a reversible residual transform; CoRe applies coherence and low-rank channel-alignment losses to iTransformer.Source: §D.3 · p. 18

• The eight methods comprise three established baselines in all three codebases and five highly cited 2026 forecasting methods available in TSFLab but not TSLib or TFB.Source: §D.4 · p. 18
• DLinear forecasts decomposed trend and seasonal parts with linear layers, while PatchTST attends over patches independently by channel and iTransformer attends across variates.Source: §D.4 · pp. 18–19
• TimeAlign aligns history and future representations using a future-reconstruction branch, while GTR retrieves full-cycle memory segments and fuses them with the input.TimeAlign inference uses the patch MLP alone; GTR follows fusion with a residual MLP.Source: §D.4 · p. 19
• TimeMosaic adapts patch size to regional information density, PHAT groups variables by FFT-derived phase and cycle, and EMAformer adds channel and phase embeddings to variate tokens.Source: §D.4 · p. 19
• The benchmark uses a chronological 70/10/20 split, train-only scaling, all channels, input 168, and horizon 24.Source: §D.4 · p. 19
• Training uses paper or official-code architecture settings, at most ten epochs, validation early stopping with patience 3, batch size 32, and seed 42.Source: §D.4 · p. 19
• Evaluation covers every test window at stride 1 and reports normalized-scale MSE and MAE.Source: §D.4 · p. 19
• Each session has a six-hour budget, including environment installation.Source: §D.4 · p. 19

D.5 Method Development Pilot

• The pilot searches for a method on ETTh1 and Weather under a fixed protocol, using validation data for search and candidate selection.It uses all channels, input 336, horizon 192, train-only standardization, seed 2021, MSE loss, Adam at lr 10^-3, at most ten epochs, patience 3, and a 90-minute, two-GPU budget capped at 12 fits and four search iterations; the frozen checkpoint is tested once.Source: §D.5 · p. 19

D.6 Evaluation Metrics

• Completion requires accepted reproduction sessions or accepted method–dataset results, with execution evidence and protocol checks; reproduction error is reported separately from acceptance.For benchmarking, acceptance also requires finite MSE and MAE and code implementing the method’s defining mechanisms.Source: §D.6 · p. 19
• Cost metrics average wall time, tool calls, and fresh tokens over attempts, including failures, then divide by accepted outcomes within each task.The denominators differ between reproduction and benchmarking, so costs are compared within each task; fresh tokens exclude cache reads.Source: §D.6 · p. 20
• Code-change metrics track added and removed Python lines and changes in functions and their cyclomatic complexity.Code metrics are averaged over sessions within each task and then across tasks with equal weight.Source: §D.6 · p. 20

D.7 API Price Assumptions

• The API-price analysis estimates normalized fresh-token charge ratios of 18.2 for Sol and 9.0 for Sonnet relative to Luna, not invoices.It assumes the same fresh-token composition across agents and five-minute Sonnet cache writes; actual charges depend on token composition, context tier, and caching.Source: §D.7 · p. 20