PaperlayerView exact revision on arXiv

Source-linked AI summary

Attenuated in-context identification in time-series foundation models: diagnosis under counterfactual inputs and repair by synthetic forced-system fine-tuning

Hong-In Won

arXiv:2610.08118v1cs.LGcs.SYeess.SY

TL;DR

This paper tests whether covariate-aware time-series foundation models can predict the output change caused by a changed future input in forced plants. Paired counterfactual tests reveal memoryless interfaces and attenuated Chronos-2 responses; dither and synthetic-system fine-tuning improve results, though limitations remain on measured plants and univariate forecasting.

  • Problem

    Whether general-purpose TSFMs can identify forced-system input–output dynamics from context remains untested against systems with known counterfactual responses.

    Source: §1 · p. 1

  • Method

    The authors compare paired counterfactual forecasts from three TSFMs with classical identification fitted to the same context, then test context dither and fine-tuning on synthetic forced systems.

    Source: §2 · pp. 2–3, §1 · p. 2

  • Results

    TimesFM-2.5 and TabPFN-TS have memoryless default interfaces, Chronos-2 attenuates dynamics, and fine-tuning largely restores response magnitude, but classical identification remains better on three measured plants.

    Source: §3 · p. 4, §5 · p. 5, §8 · p. 8

  • Takeaways & Limitations

    Useful what-if answers require an interface that represents dynamics and a prior covering the plant’s time scale; paired counterfactual inputs expose attenuation that factual benchmarks alone do not.

    Source: §8 · p. 8

    Fine-tuning reduces univariate forecasting skill: median MASE rises from 0.69 for the base model to 0.89 for the fine-tuned model.

    Source: §5 · p. 7

Abstract

from arXiv · show

Covariate-aware time-series foundation models (TSFMs) promise training-free what-if answers for instrumented plants: the change in output that a different future input would cause. We test this on forced engineering systems with exact counterfactuals, comparing Chronos-2, TimesFM-2.5 and TabPFN-TS with classical system identification fitted to the same context. Through their default covariate interfaces, TimesFM-2.5 and TabPFN-TS are memoryless: the predicted effect of an input change is a same-time function of that change ($R^2 = 1.000$ for TimesFM-2.5). Chronos-2 identifies dynamics in context but attenuates them. Its predicted effect is 0.33-0.80 of the true effect, its recovered impulse response has the wrong shape, and its error on a one-degree-of-freedom oscillator levels off at 0.57 with 8192 context samples, where ARX fitted to 256 samples reaches 0.02. Context dither at inference lowers the what-if error on all six synthetic classes without training. A 26-minute fine-tune on synthetic forced systems restores the response magnitude (sensitivity 0.83-0.96) and outperforms structure-agnostic identification on Wiener-Hammerstein and a held-out friction class. A specialised in-context identifier trained on the same data comes close, so the forced-system data carry most of the gain. On three of four measured plants classical identification remains clearly better, and the fine-tuned model loses part of its univariate forecasting skill. Paired counterfactual inputs, together with shuffled future inputs on measured records, test two properties: whether the covariate interface can represent dynamics and whether the pretraining prior covers the plant's time scale. Only the counterfactual pairs expose the attenuation.

1 Introduction

The paper tests whether covariate-aware time-series foundation models can answer what-if questions about forced plants, a capability not established by prior evaluations on systems with known counterfactuals. It diagnoses model behavior and evaluates inference-time and fine-tuning remedies against classical identification.

1 Introduction

  • The study asks whether general-purpose TSFMs identify forced-system dynamics from context, filling a gap in tests against known counterfactual responses and classical identification.The authors motivate what-if answers as changes in output caused by changes in input, needed for control, scheduling, and design exploration.Source: §1 · p. 1
  • The evaluation uses synthetic plants with exact counterfactuals and measured plants, comparing TSFMs with classical and learned identification baselines fitted to the same context.The systems include oscillators, Duffing, Wiener–Hammerstein, friction, and a held-out Bouc–Wen class.Source: §1 · pp. 1–2
  • Chronos-2 identifies dynamics but attenuates and distorts responses, while TimesFM-2.5 and TabPFN-TS have memoryless default covariate interfaces.The introduction also reports that attenuation saturates with context length and follows classical excitation requirements.Source: §1 · p. 2
  • Context dither lowers what-if error on every synthetic class, while a short fine-tune on synthetic forced systems largely removes attenuation and makes Chronos-2 competitive on nonlinear plants.A specialised identifier trained on the same data comes close; on most measured plants the adapted model still trails classical identification and loses part of its univariate forecasting skill.Source: §1 · p. 2

2 Problem and protocol

The protocol compares paired counterfactual predictions with known plant responses, alongside factual forecasting tests on measured records. It evaluates foundation models and identification baselines using matched contexts.

2 Problem and protocol

  • For each context, the protocol compares predicted and true output changes under alternative future inputs, scoring relative what-if error and sensitivity over six alternatives.Both factual and alternative forecasts begin from the same plant state, so shared free response and forecast bias cancel.Source: §2 · p. 2
  • Six synthetic system classes and a Bouc–Wen class used only for testing are evaluated alongside four measured benchmarks and a base-excitation rig.Because measured plants lack counterfactual truth, the protocol scores factual NRMSE and shuffle gain; EMPS is excluded because its periodic closed-loop record cannot test input response.Source: §2 · p. 3
  • The model comparison includes Chronos-2, TimesFM-2.5, and TabPFN-TS, plus identification baselines fitted to the same context and specialised in-context transformers.The classical comparison selects the best of ARX, output-error, N4SID, and nonlinear ARX variants; a known-structure grey-box model serves as a bound.Source: §2 · p. 3
  • The reported benchmark visual covers median what-if error at 1,024 context samples across 30 instances and six alternative inputs.The table caption distinguishes fine-tuned Chronos-2, context dither, published and retrained in-context transformers, and structure-agnostic and known-structure identifiers.Source: Table 1 · p. 4

3 Diagnosis of the covariate interfaces

The default covariate interfaces of TimesFM-2.5 and TabPFN-TS barely respond dynamically to changed inputs, while Chronos-2 uses input history but underestimates and distorts the resulting effects. More context helps Chronos-2 only partly, and excitation and noise affect its errors.

3 Diagnosis of the covariate interfaces

  • TimesFM-2.5 and TabPFN-TS have sensitivity below 0.15 and what-if error 1.00 to within 0.02 on every system with their default interfaces.TimesFM-2.5’s predicted effect is an exact same-time linear function of input change (R^2 = 1.000), unlike the plant response; adding 32 lagged inputs raises its sensitivity to 0.81–1.08.Source: §3 · p. 4
  • Chronos-2 predicts effects at 0.33–0.80 of the true magnitude and has what-if error from 0.36 on lag+delay to 0.80 on Wiener–Hammerstein.It loses to classical identification on every system at both context lengths, with the largest stated gap on 1-DOF: 0.67 against 0.01.Source: §3 · p. 4
  • On 1-DOF, Chronos-2’s error falls from 0.94 at 128 context samples to 0.57 at 8,192, while ARX fitted to 256 samples reaches 0.02.Its pulse-derived impulse-response gain is 0.41 of the true gain and its correlation with the true shape is 0.69; the context-length and pulse panels compare these dynamics with ARX and the true response.Source: §3 · p. 4, Figure 3 · p. 5
  • On 1-DOF, Chronos-2’s error is 0.50 with PRBS or wide-band excitation, but 0.95 with a single sine, compared with ARX’s 0.84.As context-output noise rises from 0% to 30% of output standard deviation, Chronos-2’s error drops from 0.73 to 0.37 while ARX’s rises from 0.00 to 0.36.Source: §3 · pp. 4–5

4 Context dither at inference

Context dither is a training-free inference remedy that reduces what-if error across all six synthetic classes and also improves factual error on three public measured benchmarks. Its benefit is not simply an ensemble effect, and the measured-plant results remain mixed.

Context dither at inference

  • On all six synthetic classes, dither lowers what-if error; it also improves Chronos-2 factual error on Wiener–Hammerstein, Cascaded Tanks, and Silverbox.On 1-DOF, error falls from 0.67 to 0.36; on Wiener–Hammerstein, from 0.80 to 0.49. On the base-excitation rig, dither worsens factual error.Source: §4 · p. 5
  • On 1-DOF, impulse-response correlation rises from 0.69 to 0.85 and gain from 0.41 to 0.84, indicating improvements in response shape and magnitude.A single dither draw gives 90% of the averaged improvement.Source: §4 · p. 5
  • Input-channel white-noise dither helps 4 of 6 classes, but bandwidth-matched dither helps none and hurts 6, consistent with benefit from wider excitation.After fine-tuning, context dither improves 0 of 6 classes and worsens 4, consistent with compensation for a deficit that fine-tuning removes.Source: §4 · p. 5
  • On measured benchmarks, dither lowers factual error on three public plants but worsens it on the base-excitation rig, and remains far from classical identification.It raises shuffle gain on the rig despite worsening factual error.Source: §4 · p. 5

5 Fine-tuning on synthetic forced systems

Fine-tuning Chronos-2 on synthetic forced systems restores response sensitivity and improves what-if error across synthetic classes. It becomes competitive with structure-agnostic identification on several nonlinear plants, but classical methods remain stronger on some measured plants and univariate forecasting degrades.

Fine-tuning on synthetic forced systems

  • Fine-tuning raises sensitivity to 0.83–0.96 and lowers what-if error on every synthetic class, including 1-DOF from 0.67 to 0.17 and Wiener–Hammerstein from 0.80 to 0.20.The 26-minute fine-tune used 3000 series from five classes; Coulomb friction was held out.Source: §5 · pp. 5–6
  • Fine-tuned Chronos-2 beats the best structure-agnostic identifier in 5 of 8 nonlinear class–length cells, ties in 2, and loses in 1.It is better on Wiener–Hammerstein and held-out Coulomb at both context lengths, while classical identification remains better on 1-DOF (0.17 against 0.01).Source: Table 3 · p. 6, §5 · p. 6
  • On measured plants, fine-tuned Chronos-2 remains worse than the best classical model on Wiener–Hammerstein and Silverbox, ties on Cascaded Tanks, and is less accurate than classical models on the base-excitation rig.On the rig, fine-tuned Chronos-2 has NRMSE 0.195 against 0.045 for OE and 0.051 for ARX.Source: Figure 4 · p. 7, §5 · p. 7
  • Training a specialised in-context identifier on the same 3000 series brings it close to fine-tuned Chronos-2, indicating that most of the gain comes from forced-system data.On Wiener–Hammerstein, its error is 0.22 against 0.20 for fine-tuned Chronos-2.Source: §5 · pp. 6–7
  • Silverbox time-scale coverage explains part of the fine-tuned model’s error: resampling threefold lowers it to 0.268, but most of its gap to ARX remains.A time-scale-randomised prior improves the original-record error to 0.443, while the short-horizon control does not help.Source: §5 · p. 7, Figure 4 · p. 7
  • Fine-tuning worsens univariate M4 forecasting: median MASE rises from 0.69 to 0.89, with worse performance on 82% of series.Adding unrelated univariate series to fine-tuning did not prevent this degradation.Source: §5 · p. 7

6 Related work

Prior work examines covariate forecasting, interventions, and in-context identification, but this paper focuses on dynamic input–output effects. Its findings connect interface design and forced-system data to whether a TSFM can identify plant dynamics from context.

Related work

  • For dynamic relations, a same-time covariate regression cannot represent delays or resonances, regardless of backbone quality.Earlier work probed static target–covariate relations [4].Source: §6 · p. 8
  • Unlike context-parroting findings for autonomous chaotic systems [6] [7], this study finds no parroting under exogenous input and locates Chronos-2’s failure in attenuation.HVAC work used a separate forced-response operator after covariate TSFMs performed poorly on intervention effects [8].Source: §6 · p. 8
  • The adaptation result aligns with synthetic system-identification work [9] [10] [11] [23]: the general forecaster needed forced-system data in its prior to act as an identifier on synthetic plants.Source: §6 · p. 8

7 Discussion

Factual accuracy tracks what-if accuracy across the synthetic forced systems, while measured-record tests distinguish whether models use inputs from whether their predicted effects are correct. The proposed checks combine paired counterfactual inputs with shuffled future inputs, with scope bounded by the tested systems and unresolved dither mechanisms.

7 Discussion

  • Spearman 0.92 correlation links factual NRMSE and what-if error across synthetic model–system cells; factual input benchmarks are informative for forced plants.For input-using models, correlations across instances of one model and system range from 0.35 to 0.96; passive-dominated buildings can have accurate factual forecasts but invalid intervention responses [8].Source: §7 · p. 8
  • Paired counterfactuals on synthetic plants test sensitivity and response shape, while shuffled future inputs on plant records test whether a model uses the input.Measured-record factual error and shuffle gain cannot establish whether the predicted effect is correct; the tested synthetic classes omit switching and strongly non-minimum-phase systems.Source: §7 · p. 8
  • Duffing and Wiener–Hammerstein results are in-distribution; held-out friction and hysteresis classes, excitations, and measured plants provide the out-of-distribution evidence.When equations are known, grey-box models remain better on Duffing, Wiener–Hammerstein, and Coulomb; the reported findings also depend on the listed package versions and engineered memory features for TimesFM-2.5.Source: §7 · p. 8
  • Output dither and white input dither improve Chronos-2, but band-limited input dither does not, and dither stops helping after fine-tuning.The pattern is consistent with context–prior mismatch and input-channel excitation bandwidth, but does not identify the internal mechanism.Source: §7 · p. 8

8 Conclusion

For the forced engineering systems studied, useful what-if answers require an interface capable of representing dynamics and a prior covering the plant’s time scale. A short synthetic fine-tune largely supplies the latter on synthetic plants, but classical identification remains better on several measured plants and fine-tuning costs some general forecasting skill.

8 Conclusion

  • A dynamic-capable covariate interface and a prior covering the plant’s time scale were needed for useful what-if answers in the forced engineering systems studied.A same-time regression cannot represent dynamics; a short synthetic fine-tune largely supplies the missing prior on synthetic plants.Source: §8 · p. 8
  • Classical identification remained clearly better on the measured Wiener–Hammerstein benchmark, Silverbox, and the authors’ rig.The fine-tuned model also lost part of its general forecasting skill.Source: §8 · p. 8
  • The synthetic generator, seeds, model versions, and training settings are given in Appendix A, and measured benchmarks are public.Code, configurations, and synthetic results are available from the corresponding author on request; the base-excitation rig records are not distributed.Source: §8 · p. 8
  • The author used AI coding and writing assistants to implement and run experiments and analysis code and to draft and edit the manuscript.The author states that they conceived the study, designed the experiments, interpreted results, and take full responsibility for the content.Source: §8 · p. 9

A Systems, inputs and versions

The appendix specifies synthetic system generation, input excitation, evaluation procedures, training, baselines, and measured-record windows. Together these settings define the tested system families and the comparisons used to assess counterfactual effects.

Synthetic systems

  • The synthetic suite includes lag+delay, 1-DOF and 2-DOF oscillators, Duffing, and Wiener–Hammerstein systems; all are sampled at one sample per unit time with zero-order-hold inputs.They use fourth-order Runge–Kutta integration with 8 sub-steps, a 600-sample burn-in, and context-mean and standard-deviation output standardisation; the same transform is applied to future outputs.Source: A Systems, inputs and versions · p. 10
  • The Coulomb class is held out from fine-tuning, whereas Bouc–Wen is used only for testing.The input includes an amplitude-dependent dead zone in the Coulomb system; Bouc–Wen models hysteresis.Source: A Systems, inputs and versions · p. 10

Inputs

  • Context inputs use filtered Gaussian noise by default, with narrow- and wide-band, multisine, PRBS, random-step, and sine alternatives in excitation studies.Evaluation instances, dither-calibration instances, and fine-tuning series use non-overlapping seeds.Source: A Systems, inputs and versions · p. 10

Versions and compute

  • The reported setup uses Chronos-2, TimesFM-2.5, TabPFN-TS, and system-identification package versions listed in the appendix, with inference and fine-tuning on one Apple M4 Max.The univariate forecasting test samples 200 M4 hourly series, forecasts 48 steps, and scales MASE by the seasonal-naive error at lag 24.Source: A Systems, inputs and versions · pp. 10–11

Evaluation procedures

  • The context-length, excitation, and noise study uses 30 instances per class, lengths 128–8192, five alternative inputs, and four output-noise levels.Impulse-response estimates come from a pulse at horizon step 8 and are compared with the system’s unit impulse response using projected gain and Pearson correlation; context masking removes 25% or 75% of output samples.Source: A Systems, inputs and versions · p. 11

Training

  • Chronos-2 fine-tuning uses 600 series per training class for 2000 steps, with narrow-band and sine contexts excluded for held-out excitation tests.The linear-only variant uses 1000 series per linear class, while the wide-prior variant time-scales training systems; ICL-TF* is retrained on segments from the fine-tuning series.Source: A Systems, inputs and versions · p. 11

Baselines

  • Classical comparisons include ARX, OE, N4SID, polynomial and simulation-error NARX, and MLP-NARX; grey-box fits use known equations for selected nonlinear systems.The grey-box fits optimize free-run context simulation error from six random starts.Source: A Systems, inputs and versions · p. 11

Measured records

  • Measured windows comprise 40 non-overlapping Silverbox and Wiener–Hammerstein windows and nine windows per Cascaded Tanks record, with overlap making its intervals optimistic.Shuffled inputs are taken from another window’s future input; intervals are bootstrap intervals over instances or windows, using 20,000 resamples for paired Table 3 and public measured-benchmark differences.Source: A Systems, inputs and versions · pp. 11–12

B Additional results

The additional results examine engineered memory covariates and replay during fine-tuning, while the appendix identifies comparisons at shorter context and across alternative inputs.

Shorter context

  • Table 4 repeats the Table 1 comparison at context length 256.The supplied caption gives the table’s scope but no cell values.Source: Table 4 · p. 12

Alternative inputs

  • Table 5 reports what-if error and sensitivity by alternative input at context length 1024, using medians over 30 instances.The caption identifies the comparison dimensions but does not provide values.Source: Table 5 · p. 12

Memory covariates

  • At context length 1024, memory covariates give TimesFM-2.5 median what-if errors of 0.40–0.87 and Chronos-2’s 1-DOF error of 0.44 versus 0.67 at baseline, but no feature set beats the best classical identifier.A causal filter bank gives errors of 0.02–0.72 for TimesFM-2.5 and 0.10–0.64 for Chronos-2.Source: B Additional results · p. 12

Replay against forgetting

  • Replay with weekly M4 series retains what-if error within 0.05 of fine-tuning alone on five of six classes, but hourly MASE worsens from 0.89 after fine-tuning to 0.97 with replay.The base model’s hourly MASE is 0.69; replay therefore does not prevent forecasting forgetting in this run.Source: B Additional results · p. 12

Related papers

Loading 2610.08118v1…