In-Context Learning for System Identification: Probabilistic Formulation, Long-Sequence Handling, and Benchmark Validation

1.1 System identification and meta learning

• Classical identification repeats model-structure selection, method choice, and hyperparameter tuning for each system, while related systems are often modeled independently.This per-system workflow does not reuse information from previously identified systems and can complicate shared-controller deployment.Source: §1.1 · p. 2
• The proposed meta model trains offline on synthetic systems, then predicts outputs from a new system’s context and query without selecting a task-specific model structure or hyperparameters.The paper frames identification around a class of systems rather than a single plant.Source: §1.1 · p. 2
• The approach requires substantial offline pre-training, and a performance gap may remain against carefully tuned classical methods when task-specific data are abundant or parsimonious control models are required.Source: §1.1 · p. 3

1.2 Related Literature

• The paper situates its approach alongside meta-learning methods such as MAML [15] and in-context learning, which predicts query outputs from a context without further training [22].Source: §1.2 · p. 3

1.3 Paper Contribution

• A probabilistically weighted ensemble handles long input–output contexts; versus patching, it improves in-distribution accuracy and reduces meta-training time, with slightly higher meta-inference time.Source: §1.3 · p. 4
• The paper reformulates synthetic data generation probabilistically to produce uncertainty-aware datasets for training classical system identification algorithms.The unified contribution builds on earlier work and extends it into a probabilistic formulation.Source: §1.3 · p. 4
• The framework is evaluated on real nonlinear benchmarks and systems outside the WH training class, with the paper reporting generalization from simulated training data to real data or different system classes.The contribution summary also describes systematic evaluation on nonlinear benchmark initiative data.Source: §1.3 · p. 4
• Input-only fine-tuning in an out-of-distribution setting achieves performance comparable to full adaptation when both system dynamics and input signals differ from training.The stated practical scope is settings where users know the excitation signals but system dynamics are uncertain.Source: §1.3 · p. 4

2 Notation

• Lowercase letters denote vectors, subscripts mark time indices, and u_a:b denotes a sequence; parenthesized superscripts index samples, while unparenthesized superscripts index subsequences.Source: §2 · p. 5
• E_p[·] denotes expectation under p, ∥u∥ the Euclidean norm, and a hat marks an estimated quantity.Source: §2 · p. 5
• N(·; μ, σ²) denotes a Gaussian probability density function with mean μ and variance σ².Source: §2 · p. 5

3.1 System Class

• The system class is modeled through a distribution over systems and datasets, with context and query sequences generated by the same system under different random inputs and initial conditions.The system distribution may reflect physical parameters informed by domain knowledge or high-fidelity simulators.Source: §3.1 · p. 5

3.2 Classic system identification vs meta modeling

• Unlike classic identification, which estimates a separate model for each system, the meta model is trained across the class and directly maps a system’s context data and query input to its output.Training minimizes expected simulation error over the dataset distribution.Source: §3.2 · p. 6

3.3 Probabilistic Prediction

• The probabilistic formulation outputs a distribution over future outputs, with the mean giving the point estimate and the standard deviation quantifying predictive uncertainty.The model uses a multivariate Gaussian with diagonal covariance and is trained to approximate the conditional output distribution by minimizing expected KL divergence.Source: §3.3 · pp. 6–7

3.4 Meta model architecture and training

• A Transformer encoder processes recurrently patched context into an implicit system representation, which a causal decoder combines with query inputs to predict distribution parameters.Training uses mini-batch gradient descent over batches of systems drawn from the system class.Source: §3.4 · p. 7

4 Handling long sequences

• Transformer attention has computational complexity quadratic in sequence length, limiting the context length the meta model can efficiently handle.Patching is applied during training, whereas ensembling is applied only after training during inference.Source: §4 · p. 8, §4.2 · p. 9

4.1 Patching

• Patching reduces the sequence length processed by multi-head attention by a factor of L, reducing meta-model training computational complexity by a factor of L^2.The recurrent network maps each non-overlapping context patch to one vector before the sequence enters the encoder.Source: §4.1 · p. 8

4.2 Ensembling

• At inference, overlapping length-m context subsequences from a longer sequence produce multiple predictions for the same query, combined by a weighted average.The weights are non-negative and chosen for an unbiased estimate with minimum variance; the variance minimization is derived under an independence assumption.Source: §4.2 · p. 9

5 Synthetic data generation for system identification

• Context-conditioned synthetic trajectories enrich scarce or weakly informative training data, but rely on the meta model being able to predict the target system from that context.The meta model predicts outputs for synthetic inputs, and the resulting trajectories are used alongside real data to estimate a parametric model.Source: §5 · pp. 10–11

5.1 Deterministic regularization

• Synthetic trajectories are added through a regularization term: deterministic regularization gives all synthetic samples equal weight, while λ balances synthetic and real data.The section says λ is chosen by hold-out validation; too much synthetic-data weight can introduce distributional bias.Source: §5 · p. 11, §5.1 · p. 11

5.2 Maximum-likelihood based regularization

• Maximum-likelihood regularization combines real-data fit with synthetic-data fit, weighting synthetic contributions inversely by their predictive variance.High-uncertainty synthetic samples have less influence, while higher-confidence predictions carry more weight.Source: §5.2 · pp. 12–13
• Before relying heavily on synthetic trajectories, practitioners can check their input spectra, amplitudes, and operating regions against real data and restrict use to supported regimes.The passage recommends regimes where the meta model is confident and synthetic inputs remain within the domain explored by the real system.Source: §5.2 · p. 14

• The evaluation includes simulation examples and system-identification benchmarks using real data.Source: §6 · p. 14
• Experiments ran on an Nvidia RTX 4090 GPU with 24GB of RAM.Source: §6 · p. 14
• The authors made the codebase, dataset-generation and training routines, and trained meta models publicly accessible for reproducibility.Source: §6 · p. 14

6.1.1 System class

• The training corpus uses Wiener–Hammerstein systems with randomly generated LTI blocks and a single-hidden-layer neural-network nonlinearity.The LTI block orders are sampled uniformly from [1, 10], and the nonlinear network has 32 hidden units.Source: §6.1.1 · p. 14
• The Wiener–Hammerstein training class is motivated by approximation results for fading-memory nonlinear systems and Volterra series, rather than chosen arbitrarily.Parallel Wiener–Hammerstein structures can exactly represent finite-order Volterra kernels, while fading-memory systems admit arbitrarily close finite-order Volterra approximations.Source: §6.1.1 · pp. 14–15
• The framework can use a simulator or physics-based model for a target domain, and lightweight adaptation with relatively little data can compensate for simulator–system discrepancies.Source: §6.1.1 · pp. 14–15

6.1.2 Meta model architecture

• Both Transformer configurations have approximately 5.5 million parameters, while the long-context patching RNN adds 16896 parameters.The Transformer uses 12 layers, 128 hidden units, and 4 attention heads in both encoder and decoder; training takes 28 hours for m = 400 and 50 hours for m = 16000.Source: §6.1.2 · p. 15
• Models are trained for context lengths m = 400 and m = 16000; only the longer-context model uses recurrent patching, dividing its context into 400 patches of length 40.Each patch is processed by a single-layer RNN before its representation is passed to the Transformer encoder.Source: §6.1.2 · p. 15

6.2 Meta model testing

• For m = 16000, the meta model uses a patching-based approach, while ensembling is used for m = 400; this allows full context data when the available training dataset at inference exceeds 400 samples.Source: §6.2 · p. 15

6.2.1 In-distribution test data

• Patching and ensembling achieve similar predictive performance on 320 in-distribution WH realizations, while patching is approximately 2 times faster at inference.Table 1 reports RMSE and inference time for predictions on query windows of length 100.Source: §6.2.1 · p. 16, Table 1 · p. 16
• Tight uncertainty bounds and low prediction error across WH systems indicate accurate predictions with well-calibrated uncertainty estimates.Figure 3 aggregates results over 320 systems and shows a representative system’s predicted mean, error, and uncertainty interval.Source: §6.2.1 · p. 16, Figure 3 · p. 16

6.2.2 Benchmarks

• On real nonlinear benchmarks, accuracy is in line with state-of-the-art methods, with competitive and sometimes state-of-the-art results on structurally out-of-distribution cascaded tanks with little real data.The benchmark test data are used only as context, without benchmark-specific training.Source: §6.2.2 · pp. 17–18, Table 2 · p. 18
• On the CED benchmark’s second output trajectory, prediction errors remain small and within the uncertainty band for most time steps, despite training only on simulated systems.The figure compares true output, predicted mean, simulation error, and the 99.7% uncertainty interval.Source: Figure 4 · p. 18, §6.2.2 · p. 19

6.2.3 Model adaptation and fine-tuning

• For CSTR, RMSE drops from 1.249 zero-shot to 0.242 with input-only adaptation and 0.115 with input and system-class adaptation, near the noise floor σnoise = 0.1.The scratch-trained model overfits; its best validation model has a test RMSE of 0.136 and takes about 4000 iterations to reach best validation performance, versus 1000 for fine-tuning.Source: §6.2.3 · pp. 19–20

6.3 Synthetic data generation for CED benchmark

• 0.70 and 0.88 were the median R2 scores for the first and second outputs without synthetic data, versus 0.88 and 0.93 with synthetic data.Figure 6 compares the two conditions over 100 Monte Carlo runs.Source: Figure 6 · p. 21, §6.3 · p. 21
• From 300 training samples, the meta model generated 200 synthetic samples for dynoNet training; the CED dataset contained only 400 samples.The same 100 samples were used for validation and regularization tuning, and the meta model had been pretrained on simulated WH models without target-specific fine-tuning.Source: §6.3 · pp. 20–21
• R2 variability was lower with synthetic data, although variation then also reflected the selected regularization parameter and generated trajectories.Without synthetic data, variability came from dynoNet weight initialization; with it, those additional sources also contributed.Source: §6.3 · p. 21

7 Conclusions

• The meta model predicts a system’s outputs from its context and query inputs without estimating a separate model for that system.It is trained offline on synthetic data and bypasses traditional system-identification algorithm and hyperparameter selection.Source: §7 · pp. 21–22
• Sequence patching and ensembling enable long-context processing while avoiding Transformers’ quadratic attention complexity.The meta model can also generate synthetic data for classical system identification, drawing on information learned during pretraining.Source: §7 · p. 22
• Numerical results show generalization from simulated training data to real data or systems from a different class, potentially with quick adaptation.Source: §7 · p. 22
• Future directions include both scaling meta models and distilling lighter architectures, as well as exploring alternatives to Transformers and adapting open models.The proposed possibilities include Structured State Space and diffusion models, and low-rank adaptations of LLaMA or DeepSeek.Source: §7 · p. 22
• The approach requires substantial offline computation, is relatively heavy at inference, and may trail classical methods with abundant high-quality task-specific data.Its predictions also lack the immediately interpretable parametric representation that some analyses and control-design tasks require.Source: §7 · p. 22

A Minimum-variance weights for ensembling

• For a specific time step k, the derivation minimizes µk by selecting weights.Source: A Minimum-variance weights for… · p. 26
• The weights are obtained from an optimization problem subject to a constraint.Source: A Minimum-variance weights for… · p. 26
• Lagrange multipliers yield an augmented Lagrangian, and substituting γ gives the optimal weights in (15).Source: A Minimum-variance weights for… · p. 26