Source-linked AI summary
Forgetting While Remembering, an Invariant Online Data-Driven Predictive Control Formulation
Alessandro Chiuso, Florian Dörfler, Keith Moffat
TL;DR
Online DPC for time-varying systems faces low-SNR challenges, motivating a method that uses a prior while adapting how it forgets old data. The paper combines an adaptive Kalman-filter ARX estimator with FCE-based control and reports tracking experiments on a time-varying second-order system.
Problem
Online DPC for linear time-varying systems must adapt its forgetting rate, since excessive reliance on recent data can create low-SNR training and stability or robustness problems.
Source: §I · p. 1Method
The method adapts an ARX Kalman-filter process equation while keeping the prior invariant, then uses posterior parameter mean and variance in Bayesian DPC optimization.
Source: §I · pp. 1–2Results
Tracking experiments report that turning off FCE regularization leads to instabilities, and running-average cost comparisons again show the importance of regularization.
Source: §VI · p. 5Takeaways & Limitations
The formulation estimates ARX parameters, their uncertainty, and the process equation together, while maintaining the parameter prior over time.
Source: §VII · pp. 5–6
Abstract
from arXiv · showhide
Low signal-to-noise ratio (SNR) data is a core challenge of online Data-Driven Predictive Control (DPC) for linear, time-varying systems. This paper proposes a Bayesian, online DPC framework based on autoregressive models with exogenous inputs (ARX) that uses an externally-provided prior, which encodes inductive bias such as smooth system dynamics and stability, to safeguard performance when SNR is low. The posterior estimate of the ARX parameter is propagated forward in time using a Kalman filter with a state equation defined by an adaptation-rate hyperparameter, which is adjusted online to track the rate at which the underlying system dynamics evolve. The Kalman filter's posterior mean and covariance determine the DPC's Final Control Error cost function, which is the posterior expectation of the quadratic cost function. Critically, the Kalman filter's process equation is chosen so that, regardless of the hyperparameter adaptation, the prior distribution is invariant over time. Thus, while old data is forgotten, the prior is not. DPC tracking experiments on a time-varying second-order system demonstrate the efficacy of the proposed method.
I. INTRODUCTION
Online DPC for time-varying systems must balance tracking changing dynamics against the low-SNR risks of relying too heavily on recent data. The paper proposes an adaptive Bayesian ARX estimator that changes how data are forgotten while keeping the prior invariant, and uses its posterior in an FCE-minimizing controller.
I. INTRODUCTION
- When dynamics change slowly, heavy reliance on recent data can cause low-SNR training, stability loss, and robustness problems; when they change quickly, old data can misrepresent the system.The appropriate adaptation rate is system-dependent and time-varying.Source: §I · p. 1
- A prior encoding inductive bias such as smoothness or stability can anchor Bayesian estimates and reduce measurement-variance impact in low-SNR data.The cited adaptive predictive-control approaches do not use an adaptive forgetting factor and remain susceptible to low-SNR issues.Source: §I · p. 1
- The proposed ARX estimator adapts its process equation to forget older data while preserving an invariant prior distribution, unlike approaches that adapt process noise.The paper describes this invariant-prior, adaptive-process-equation estimator as novel in adaptive filtering and control.Source: §I · p. 1
- The controller minimizes the posterior expected quadratic cost, using the Kalman filter’s ARX-parameter mean and variance; its single-step predictor avoids closed-loop bias issues.The formulation builds on the FCE and sufficiency result introduced in.Source: §I · pp. 1–2
II. PROBLEM STATEMENT
The paper formulates online control of a stochastic, linear time-varying system through a data-driven predictor and a receding-horizon objective. At each time, the controller chooses future inputs to minimize the expected cost conditional on past data and an externally provided parameter prior.
II. PROBLEM STATEMENT
- The setup considers a stochastic, linear time-varying system with scalar input and output, presenting the SISO case while noting that the construction extends to MIMO systems.The predictor uses a constant-over-time chosen length, selected using AIC as discussed in.Source: §II · p. 2
- The receding-horizon problem selects future inputs over control horizon T to track a desired output sequence while minimizing the ground-truth cost.In the data-driven setting, past data are used to predict future outputs as a function of future inputs.Source: §II · p. 2
- Rather than minimizing a certainty-equivalent cost, the formulation minimizes the conditional expected cost, called the Final Control Error (FCE) in.The time-varying case uses an approximation decomposed into a certainty-equivalent cost and an uncertainty regularization term.Source: §II · p. 2
- At each time, the control action minimizes FCE given past data and the prior variance on predictor coefficients, under an externally provided Gaussian prior for the ARX parameters.The prior is zero-mean for notational simplicity; the method extends directly to a nonzero prior mean.Source: §II · p. 2
III. FORGETTING WHILE REMEMBERING: AN INVARIANT ADAPTATION MODEL
The adaptation model is motivated by two extremes: a white-noise parameter model adapts quickly but uses only the latest measurement, while a time-invariant model weighs all data equally. The proposed method interpolates between these cases to adapt to changing evolution rates.
III. FORGETTING WHILE REMEMBERING: AN INVARIANT ADAPTATION MODEL
- A white-noise parameter model supports quick adaptation to system changes but uses only the most recent measurement.In this extreme, the current parameter provides no information about the next one.Source: §III · p. 2
- A time-invariant parameter model weighs all data equally but does not adapt effectively when the underlying system changes.In this extreme, the parameter remains constant over time.Source: §III · p. 2
- The proposed method interpolates between the white-noise and time-invariant extremes in a data-driven, adaptive manner to adjust to time-varying evolution rates..Source: §III · p. 2
A. System Prior
The system prior uses a process equation whose adaptation parameter controls how quickly ARX parameters vary, while the prior distribution remains invariant over time. This allows the model to forget older data without forgetting the externally supplied prior.
A. System Prior
- The parameter α controls the process equation and represents the rate of parameter variation; α is estimated online so adaptation speed is learned from data.Large α corresponds to slowly varying dynamics, with α = 1 yielding the time-invariant limit.Source: A. System Prior · pp. 2–3
- The prior variance encodes parameter correlations and can assert properties such as smoothness or stability.This gives the invariant prior a role beyond initializing parameter estimates.Source: A. System Prior · pp. 2–3
- The process equation preserves the zero-mean Gaussian prior with variance Q at every time when the initial parameter and process noise have that same distribution.The proof establishes the result by the recursive mean and variance equations.Source: A. System Prior · p. 3
B. Recursive, Bayesian ARX parameter estimation
The ARX parameter is modeled in a time-varying linear state-space form and estimated recursively with a Kalman filter. Its prediction and measurement updates track the parameter mean and covariance from input-output data.
B. Recursive, Bayesian ARX parameter estimation
- The Kalman filter recursively estimates the conditional mean and variance of the ARX parameter using the observed data.The measurement noise represents one-step-ahead prediction error, while process noise models parameter evolution.Source: B. Recursive, Bayesian ARX parameter… · p. 3
- The prediction and filtered estimates use data through t−1 and t, respectively, with a covariance for each estimate.The process-noise covariance Q is also the parameter variance before measurements.Source: B. Recursive, Bayesian ARX parameter… · p. 3
- For the paper’s SISO formulation, the measurement and innovation variances are scalars, while parameter covariances are matrices.The matrix dimensions are 2^ρ × 2^ρ.Source: B. Recursive, Bayesian ARX parameter… · p. 3
- With a zero-mean prior, the process equation pulls the predicted parameter estimate toward zero, while the measurement update moves it to describe observed data.The adaptation parameter α lies in [0, 1].Source: B. Recursive, Bayesian ARX parameter… · p. 3
C. Recursive Final Control Error
The method recursively builds a T-step prediction from a one-step ARX predictor, updating its data window with candidate inputs and predicted outputs. It uses the resulting prediction mean and covariance to minimize the Final Control Error through a strictly convex quadratic program.
C. Recursive Final Control Error
- The T-step predictor recursively applies the one-step ARX predictor, appending each candidate input and predicted output while discarding the oldest data pair.The predictor uses the predicted ARX coefficients.Source: C. Recursive Final Control Error · p. 3
- The candidate input sequence affects the multi-step output prediction mean and the propagated uncertainty in that prediction.The covariance term captures how ARX coefficient uncertainty propagates into the multi-step prediction.Source: C. Recursive Final Control Error · p. 3
- The control sequence is obtained by minimizing the FCE with an unconstrained, strictly convex quadratic program.Its unique solution follows from the linear system given by the first-order optimality condition.Source: C. Recursive Final Control Error · pp. 3–4
IV. LEARNING TO FORGET: SELF-TUNING OF THE ADAPTATION SPEED
The adaptation rate α is tuned online by maximizing the marginal likelihood of observed data after marginalizing over the ARX parameter. To handle noisy single-sample losses, the method exponentially averages gradient information and uses a Newton/Hessian-based update.
IV. LEARNING TO FORGET: SELF-TUNING OF THE ADAPTATION SPEED
- The adaptation rate α is estimated by maximizing the marginal likelihood of observed data after marginalizing over the ARX parameter.This marginalization yields a Gaussian one-step predictive distribution for the observation.Source: §IV · p. 4
- Online tuning uses a single-sample loss, a noisy Monte Carlo estimate of the expected one-step negative log-likelihood, so the method exponentially weights gradient information.The exponentially filtered gradient is the gradient of an exponentially weighted linearized surrogate; averaging past samples mitigates single-sample variability at the cost of temporal smoothing.Source: §IV · p. 4
- The exponentially weighted gradient and Hessian are updated using forgetting factor β.The hyperparameter update uses a Newton/Hessian-based step so α can adapt quickly in rapidly time-varying applications.Source: §IV · p. 4
V. PROPOSED DATA-DRIVEN PREDICTIVE CONTROL
At each time step, the proposed algorithm chooses a future input sequence by minimizing the FCE, applies its first action, then updates α and the Kalman filter. The prior covariance Q can encode smoothness or stability, and the FCE is the posterior expectation of the control cost.
V. PROPOSED DATA-DRIVEN PREDICTIVE CONTROL
- At each time step, the algorithm minimizes FCE over a future input sequence, applies the first action, updates α, and iterates the Kalman filter.The loop then advances time and repeats.Source: §V · p. 4
- The test system’s poles and zeros transition between endpoint configurations through a scheduled parameter that ramps up and then back down.The schedule holds at each endpoint before and after the ramps.Source: §V · p. 5
- The FCE is the posterior expectation of the control cost, parameterized by the predicted ARX coefficient mean and covariance.The prior on the ARX coefficients has covariance Q, which can encode smoothness or stability.Source: §V · p. 5
VI. RESULTS
The tracking experiments examine the proposed controller on a time-varying system and show that regularization matters: instability arises when it is turned off. The cost comparison also supports the importance of regularization.
VI. RESULTS
- The test system interpolates between a low-pass system with roughly 40 dB static gain and a resonant system with roughly 5 dB gain.The dynamics follow a hold–ramp-up–hold–ramp-down–hold scheduling profile.Source: §VI · p. 5
- The externally supplied DC prior favors stable, smooth impulse responses.The prior hyperparameters are selected by averaging variances from 100 uniformly sampled hyperparameter sets.Source: §VI · p. 5
- The tracking plots show instability without regularization, and the authors report that regularization is important.Figures 1 and 2 show reference and controlled-output traces, including a zoom around time 800.Source: §VI · p. 5, Figure 1 · p. 5, Figure 2 · p. 5
- The running-average cost comparison between FCE and certainty-equivalent objectives likewise supports the importance of regularization.Figure 3 reports exponentially weighted running averages of the cost.Source: §VI · p. 5
VII. CONCLUSION
The conclusion presents an online DPC method that propagates ARX parameter beliefs while keeping the prior invariant. Its control action uses parameter estimates and uncertainty, and the appendix explains how adaptation-rate updates respond to predictive fit and variance.
VII. CONCLUSION
- The parameter-propagating process equation maintains the system-parameter prior over time, while the method estimates ARX parameters, their uncertainty, and the process equation.The conclusion describes simulation results as promising and calls for further investigation.Source: §VII · pp. 5–6
APPENDIX I INTUITION FOR THE PROCESS EQUATION ADAPTATION
- The mean-fit term adjusts α according to how it changes the predicted output, driving updates toward reduced one-step prediction error.For example, when the prediction error is positive and increases with α, gradient descent decreases α.Source: Appendix I · p. 6
- The variance-fit term adjusts α so the one-step predictive variance better matches the observed innovation magnitude.When the innovation is larger than predicted and predictive variance decreases with α, gradient descent decreases α, favoring faster parameter variation.Source: Appendix I · p. 6
A. Proof of Lemma 1:
The supplied proof passages describe likelihood-based gradient and expected-Hessian steps for Lemma 1, an induction argument for Lemma 3, and a constrained reparameterization for adapting α.
A. Proof of Lemma 1:
- The derivation takes the negative log-likelihood, drops additive constants, and simplifies expected-Hessian terms before giving scalar formulas.The proof passage also identifies a gradient with respect to α.Source: A. Proof of Lemma 1 · pp. 6–7
B. Proof of Lemma 3:
- The Lemma 3 proof differentiates the smoothed loss and obtains the stated result by induction, initializing the accumulated gradient at zero.It also takes the gradient of the alternative loss expression with respect to ν.Source: B. Proof of Lemma 3 · p. 7
APPENDIX III RE-PARAMETERIZATION OF α
- Rather than update α directly, the method updates η through a logistic reparameterization and projects Newton steps onto [−1, 1].For the slowly varying systems considered, α is restricted to [0.9, 1]; faster variations may require a smaller lower bound.Source: APPENDIX III RE-PARAMETERIZATION OF α · p. 7