PaperlayerView exact revision on arXiv

Source-linked AI summary

Forecasting Multiple Observables with SCROLL: Score-Trained Uncertainty for Stochastic Dynamics

Pavel Prochazka

arXiv:2608.25898v1cs.LG

TL;DR

Forecasting stochastic systems involves multiple observables with distinct likelihoods, while standard multi-task weighting does not address likelihood misspecification. SCROLL-MT composes task-specific likelihoods and free-routed last-layer beliefs on a shared backbone, recovering the analytic OU law and improving variance-aware performance on heteroscedastic systems and real air-quality data. The study remains scoped to a small set of systems, tasks, and lead times, and joint training can suffer task interference.

  • Problem

    Multi-task forecasting needs likelihoods for several observables, but weighted-loss recipes address scaling separately from whether each likelihood can represent the target.

  • Method

    SCROLL-MT composes per-task likelihoods and Gaussian free-routed last-layer beliefs on one jointly trained shared backbone, without outer unit-balancing search.

  • Results

    SCROLL-MT recovers the analytic OU predictive law, tracks Lorenz-63 state-dependent variance, and achieves best or statistically tied single-run performance across the real-series columns.

  • Takeaways & Limitations

    The results support composed likelihoods and freely routed beliefs as useful for uncertainty-aware multi-observable forecasting within the tested SDE and real-series settings.

  • Takeaways & Limitations

    The study covers two controlled low-dimensional SDE systems, one small real series, K=3 tasks, and one lead time per run, and joint training can suffer task interference.

Abstract

from arXiv · show

Forecasting a stochastic dynamical system rarely means a single number: one wants several observables---future state, threshold event, regime label---each with its own likelihood. Standard multi-task recipes balance per-task losses, tuned or learned. We instead compose the observables' likelihoods in per-task free-routed last-layer beliefs on a shared backbone; this absorbs unit-dependent loss scaling into likelihood parameters learned in the same gradient pass. Stochastic dynamics supply what static benchmarks cannot: computable ground truth for the predictive variance. Results land where theory puts them: on the well-specified, homoscedastic Ornstein--Uhlenbeck process the learned predictive law recovers the analytic kernel and correctly specified baselines tie. On heteroscedastic systems (stochastic Lorenz-63, real air-quality data) the belief's input-dependent variance separates: best single-run NLL on the state and regime tasks, calibration matched only by arms whose NLL it beats, at a fraction of the tuned grids' cost. On the real series the state margin holds across five rolling origins.

1 Introduction

SCROLL addresses multi-observable forecasting by retaining each observable’s likelihood and free-routed last-layer belief on a shared backbone. Its composed objective removes outer unit-balancing searches while preserving likelihood-specific uncertainty modeling.

  • Motivation: Multi-task recipes weight per-task losses, but cannot repair a misspecified likelihood head through weighting alone.A cross-entropy head fails to represent an offset barrier boundary at any tuned or learned weight; a threshold likelihood repairs it through its cutpoint.
  • Approach: SCROLL gives each observable its own observation model and Gaussian last-layer belief while jointly training one shared backbone.The construction keeps one local log-partition term per model factor.
  • Contribution: SCROLL-MT composes K free-routed single-task objectives factor by factor on one backbone.The composition is defined as the sum of K single-task SCROLL objectives coupled through the shared representation.
  • Contribution: The composition replaces outer weight search with O(K) likelihood parameters learned in one gradient pass at equal per-task convention.The stated properties also include score-optimal population behavior and scale equivariance without weight retuning.

2 Free routing and likelihood composition in the last layer

SCROLL-MT composes task-specific predictive likelihoods and free-routed last-layer beliefs through a shared backbone. Its proper-score objective supports score-optimal predictive laws and scale-equivariant task handling, while task interference and preference trade-offs remain scope boundaries.

  • Objective: Each task retains its own likelihood, Gaussian last-layer belief, and prior, while only the backbone is shared.The task-specific likelihood parameters, means, covariances, and precisions are optimized jointly with the backbone.
  • Composition: SCROLL-MT is term by term the sum of K SCROLL objectives coupled only through the shared backbone.For fixed backbone features, each data term is an exact strictly proper log score whose population optimum is the true conditional law when representable.
  • Score optimality: For Gaussian tasks, the unrestricted population optimum targets conditional residual variance, with predictive variance determined by the likelihood and belief.The score-optimal variance is the conditional residual variance about the model mean.
  • Implicit task weighting: The composed data terms enter in nats with unit coefficient per declared task, replacing unit-correction weights with learned likelihood parameters.The parameters (ηk, αk) are learned in the same pass as the beliefs and network.
  • Scale equivariance: Rescaling one Gaussian task’s targets preserves the minimizer over the backbone through a parameter-only map, unlike fixed weighted raw losses.Fixed weighted losses require retuning other-task weights under target rescaling.
  • Scope boundary: When tasks conflict, composition fixes units but does not select a unique preference on the trade-off surface.Joint training need not preserve per-task optimality because of an interference channel.

3 Experiments

Experiments test SCROLL-MT on OU, stochastic Lorenz-63, and Beijing PM2.5 with state, event, and ordinal observables. The results support analytic recovery on OU, state-dependent variance tracking on Lorenz-63, and strong single-run performance on the real series, within a deliberately limited scope.

  • Experimental setup: OU provides an analytic, state-independent variance target, while Lorenz-63 supplies heteroscedastic Monte-Carlo ground truth and PM2.5 supplies the real-data arm.Lorenz-63 conditional standard deviations at Δ=1 span p10/p90 values of 1.5/8.4; PM2.5 has no computable V⋆.
  • OU: Within 2% at every lead, OU total predictive standard deviation matches the analytic V⋆(Δ) across 10 seeds.The OU result is shown as ±1 standard deviation over seeds.
  • Methods: SCROLL-MT uses Gaussian, thresholded-Gaussian, and ordinal-probit heads with None, Diag, or Full belief covariance, without unit correction or outer weight search.Baselines include fixed and grid-tuned MAP, learned-weight Kendall arms, and input-dependent MLE σ(x) arms.
  • Comparisons: SCROLL-MT matches or beats the better fair grid per corpus at a fraction of its compute.The event head discriminates mainly on OU; elsewhere event NLLs lie within a 0.05-nat band, so claims focus on state and regime heads.
  • Lorenz-63: Pearson r=0.61 (Diag) and 0.85 (Full) measure Lorenz-63 predicted-versus-Monte-Carlo conditional variance tracking.Rank correlations are 0.56 and 0.66, while Full lies within 3% of residual variance about its own mean.
  • Real data and scope: On the real series, the state margin survives five rolling origins, while sharing is free on OU, costly on Lorenz, and costless on PM2.5.The joint model is favored on all three heads across the five real-series origins, without resolving the comparison statistically.

4 Conclusion and outlook

This deliberately scoped first study evaluates composed likelihoods with freely routed beliefs on two controlled SDE systems and one small real series. Within that scope, the method performs best or ties across single-run comparisons without outer weight search, with the real-series state margin surviving five rolling origins.

  • SCROLL-MT is evaluated on two controlled low-dimensional SDE systems, one small real series, K=3 tasks, and one lead time per run.
  • The SDE settings provide computable V⋆(x), enabling comparison of trained predictive variance with the exact target.
  • On the real data, composed likelihoods are best or statistically tied in every column among single-run methods, while Ens-5 is best overall.
  • The real-series state margin persists across five rolling origins, and the method comes within 0.01 nats of the best tuned grid on OU.
  • The reported gains are attributed to the observation model and freely routed belief rather than the weighting axis, without an outer weight search.

A SCROLL in brief

SCROLL derives a composed predictive-score objective from a factor graph, using per-task Gaussian last-layer beliefs whose routing determines the predictive variance. Free routing removes the pinned-variance restriction and recovers expressible residual heteroscedasticity, while the score optimum remains conditional on the model and population setting.

  • SCROLL derives its loss from each factor’s local normaliser, producing a composite predictive-score objective plus a prior-overlap term.
  • For each task, the model uses an observation likelihood and Gaussian last-layer belief on a shared deterministic backbone.
  • For Gaussian heads, the predictive variance is V_n = σ_obs^2 + ψ_n^TΣψ_n, while discrete heads use probit constructions with belief-induced scale widening.
  • Free routing makes (µ, Σ) direct optimisation variables, allowing the objective to choose the belief instead of imposing a posterior-prescribed binding map.
  • The belief term ψ^TΣψ is a score-trained conditional spread, not posterior epistemic variance, and need not contract as sample size grows.
  • The score-optimal population variance is the conditional residual variance V⋆(x), but free routing can recover only the heteroscedasticity expressible by the variance family.

B Proofs of the formal statements

The formal results establish that the multi-task objective decomposes into taskwise SCROLL objectives, preserves per-task score optimality at fixed backbone, and learns likelihood parameters in the same differentiable objective. Scale equivariance holds for the ideal covariance families, while the implemented covariance floor is an external caveat measured not to bind in the tested range.

  • Lemma 1 groups the multi-task objective into K single-task SCROLL blocks coupled only through the shared backbone ψ.
  • At fixed ψ, minimising the composed objective over task-specific parameters minimises each task block separately, so per-task population statements apply block by block.
  • The Gaussian data term uses predictive variance V_n, with V_n^-1 occupying the role of a MAP squared-error weight and log V_n keeping it finite.
  • Under target rescaling, the composed objective changes only by a parameter-independent constant, preserving the minimiser without retuning weights.
  • A fixed weighted sum of raw losses instead requires λ_j → c^2λ_j to recover the original minimiser after rescaling task k.
  • Proposition 1 concerns ideal covariance families; the implemented εI floor lies outside them but does not bind in the tested range.

C Related work

The paper situates its approach among grid-searched loss weighting, learned uncertainty weighting, and gradient surgery. It distinguishes composition from these weighting and optimisation approaches while noting that likelihood-based weighting predates SCROLL.

  • Prior multi-task approaches include grid search, learned uncertainty weighting, and gradient surgery, with gradient surgery treated as orthogonal to composition.
  • Kendall et al. already derive a no-weights conclusion for point-estimate MAP training from the likelihood view, and SCROLL extends the setting to richer observation models and beliefs.

D Benchmark generation and training details

The benchmarks combine analytically tractable OU dynamics, heteroscedastic Lorenz-63 simulations, and real Beijing PM2.5 windows, with Monte Carlo variance probes and a common training protocol. SCROLL-MT hyperparameters are learned in one pass, while baseline weighting and validation procedures are explicitly calibrated for mixed units.

  • Benchmark generation: OU uses an exact analytic transition density, making its conditional variance V ⋆(∆) available without discretisation bias.The process is simulated from its analytic transition density on a grid.
  • Benchmark generation: Lorenz-63 uses 65 noisy chains, 6,500 windows, and Euler–Maruyama integration at dt = 0.005.Chains start near (1, 1, 25), burn in for 2,000 steps, and use 100-step decorrelation gaps.
  • Benchmark generation: Lorenz ground truth re-simulates 200 ensemble members per state, with V ⋆ spanning an estimated 284× range across test states.The averaged figure uses 2,000 attractor states; Monte Carlo targets carry simulation error, so precise correlations and ranges are not inferential quantities.
  • Benchmark generation: Beijing PM2.5 contributes 1,680 non-overlapping 24-hour windows after handling short gaps by linear interpolation and dropping windows touching longer gaps.Inputs include recent PM2.5, meteorology, calendar features, and wind direction; the state target is log(1+PM2.5) at t+24 h.
  • Training protocol: All methods use standardised inputs, centred regression targets, Adam at learning rate 0.03, up to 5,000 full-batch steps, and summed-validation-NLL early stopping.Every objective also includes a backbone ℓ2 term.
  • Training protocol: SCROLL-MT learns prior precision and likelihood parameters in the same gradient pass, whereas MAP baselines use a 49-run λ grid and Kendall arms learn task log-scales.The fair grid is selected by the unweighted validation sum MSE + NLL1 + NLL2, whose mixed units make regression dominate according to target spread.

E Detailed results

SCROLL-MT’s detailed results support likelihood-specific routing, scale-equivariant training, and input-dependent uncertainty, while exposing covariance-family and optimisation boundaries. It recovers the OU kernel, improves heteroscedastic forecasting under suitable representations, and preserves real-data margins after standardisation and rolling-origin evaluation.

  • Likelihood families: 67% CE accuracy on the OU barrier event trails the 80% majority rate, whereas threshold-likelihood arms recover ≈0.27 nats and Lorenz favours task-specific likelihoods.The CE deficit comes from a bias-free head pinned at x=0; adding one bias lifts standalone CE to 89%.
  • Cost and tuning: 13–18 s versus 156–471 s per seed separates SCROLL-MT from tuned grids, while its single pass also learns Gaussian scales, probit cutpoints, and αk.The self-tuned run remains within 0.006 nats of grid-selected regression NLL and within 0.017 on every head.
  • Scale equivariance: 61× versus a 63× target-variance ratio shows the effective squared-error weight adapting continuously without the grid’s scale search.The selected unstandardised λ1 spans [0.01, 10], while standardisation moves the argmin to λ=(30, 30) on 9 of 10 CE seeds.
  • Scale equivariance: 0.000 ± 0.000 nats remains after mapped warm-starting at c=8, closing the fixed-budget gap attributed to optimisation.Initialisation mapping removes 39% of the gap, a 4× step budget removes 24%, and together they remove 76%.
  • OU anchor: Within 1.5% of the OU entropy floor for ∆≥0.25, the learned predictive standard deviation stays within 2% of V⋆ and σobs within 4%.All routes tie on OU, showing that freely routed variance does not invent spread on the homoscedastic anchor.
  • Covariance and backbone effects: Full improves over the pinned route by 0.137 to 0.314 nats across backbones, whereas Diag’s benefit depends on layer normalisation and representation.On unnormalised tanh, Diag cannot reach V⋆; the limitation is attributed to the covariance family rather than routing.

F Open directions

The paper identifies open questions around scaling SCROLL beyond three tasks, modeling cross-observable dependence, and separating shared-backbone interference from optimization effects. It also outlines extensions to joint horizon modeling, richer systems, sequential filtering, and broader real-series validation.

  • Scaling in the number of tasks: At K=3, the 49-run grid remains feasible, but whether SCROLL's predicted O(K) advantage grows with task count is unresolved.The task inventory itself remains a design decision because duplicating an observable doubles its data terms.
  • Jointly trained horizon sets: Treating lead times as composed tasks could produce one σ(∆) curve and make OU cross-horizon monotonicity directly testable.The current experiments use separate runs for each lead time.
  • Interference in the shared backbone: Joint training can incur a 0.06–0.07-nat Lorenz sharing cost, but the study does not distinguish finite-width feature limits from competing-gradient optimization.Widening the backbone at a fixed task inventory is proposed to separate these mechanisms.
  • A joint observation model: A diagonal joint Gaussian omits residual correlations between observables, while a full covariance model recovered ρ=0.80 at ρ=0.8 and improved joint test NLL by 0.163 ± 0.009 nats.The limitation is especially relevant when one observable is a deterministic function of another, as in the Lorenz event and regression targets.
  • Validation scope: The study deliberately uses two low-dimensional SDEs, one small real series, K=3 tasks, and one lead time per run, leaving broader empirical validation unresolved.Five rolling origins strengthen the real-data check, but they overlap heavily and do not resolve whether sharing helps across independent series.
  • The sequential cavity and adaptive filtering: A sequential-cavity formulation could connect SCROLL to prequential prediction and, in the linear-Gaussian case, recover the Kalman filter with adaptive noise-covariance updates.The shared cavity is described as an exchangeable approximation whose order-dependence disappears for temporally ordered data.
Loading 2608.25898v1…