PaperlayerView exact revision on arXiv

Source-linked AI summary

Information Geometry of Message Passing

Mykola Lukashchuk, Kyrylo Yemets, Alex Ledbetter, İsmail Şenöz

arXiv:2608.15922v1cs.LGcs.AI

TL;DR

Approximate Bayesian inference needs practical ways to handle high-dimensional models beyond tractable special cases. This paper localizes variational inference’s natural-gradient stationarity into natural-gradient message passing, which improves accuracy and uncertainty calibration when uncertainty persists around non-conjugate factors.

  • Problem

    High-dimensional Bayesian inference is often computationally intractable, motivating approximate methods for models beyond special tractable classes.

  • Method

    NGMP derives edge-local updates from Bethe free energy by Fisher-projecting each exact BP log-message onto the receiving edge’s exponential family.

  • Results

    When uncertainty persists at inputs to non-conjugate factors, NGMP estimates marginals more accurately than VMP and improves predictive calibration while retaining competitive point error.

  • Takeaways & Limitations

    Retaining the receiving family’s representable component of exact messages benefits uncertainty calibration and prediction when approximate updates are repeatedly composed.

  • Takeaways & Limitations

    A fully rigorous ambient-space treatment would require a non-parametric density-manifold formulation.

Abstract

from arXiv · show

We show that the natural-gradient stationary condition of variational inference has an edge-local form on a Forney-style factor graph. We start from the Bethe free energy and constrain a selected edge marginal to an exponential family. At a stationary point, the natural parameter of that edge equals the sum of two projected messages, one from each incident factor. Each projected message is the natural-gradient projection of the exact belief-propagation log-message at the current receiving marginal, or equivalently, the gradient of its expectation in the so-called mean coordinates. We call the resulting scheme natural-gradient message passing (NGMP). The rule is local; each edge may carry its own exponential family, and the message a factor sends depends on the marginal that receives it. Compared with variational message passing, NGMP keeps the part of the exact message that the receiving family can represent instead of averaging the factor under the neighboring beliefs. The two coincide when the uncertainty on the edges entering a non-conjugate factor vanishes, and NGMP is more accurate when that uncertainty persists, for example, along a partially observed latent chain or when parameters are filtered through successive data batches. Experiments on Poisson smoothing, heteroskedastic regression, and hourly ETTh forecasting confirm this and show that the gain appears mainly in uncertainty calibration.

1 Introduction

Bayesian inference is often computationally intractable, so variational inference replaces exact integration with optimization over a tractable family. This paper connects global natural-gradient stationarity with local factor-graph message updates through natural-gradient message passing.

  • Motivation: Bayesian inference requires evaluating a posterior normalizing constant that is often a high-dimensional integral.Closed-form solutions are limited to special model classes, while exact and worst-case guaranteed approximate inference are NP-hard.
  • Variational inference: Variational inference replaces exact integration with optimization over a tractable candidate family, whose limitations depend on the chosen family and constraints.The global route uses one approximating distribution for the entire latent state, whereas the local route converts the problem into local computations.
  • Local message passing: Factor graphs enable local inference through edge-based belief-propagation messages, yielding exact edge marginals on trees and Bethe or structured variational approximations on loopy graphs.Locality does not guarantee finite-dimensional messages when non-conjugate factors produce functions outside the supported message family.
  • Contribution: The paper localizes the global natural-gradient stationary condition by starting from the Bethe free energy and constraining selected edge marginals to exponential families.This establishes the setup for a message-level rule on a factor graph.
  • Natural-gradient message passing: Natural-gradient message passing uniquely preserves the exact BP message information visible to the receiving edge’s form-constraint family.Unlike VMP and related variants, which use variational surrogates and may additionally project the resulting message, NGMP projects the exact message information at the receiving marginal.
  • Empirical motivation: When uncertainty entering a non-conjugate factor cannot be reduced, projected messages estimate marginals more accurately than VMP and projective variants at comparable complexity.Hourly ETTh forecasting indicates that predictive scale can collapse despite competitive point error, motivating improved uncertainty calibration.

2 Background

This section introduces Forney-style factor graphs and belief propagation as a local way to compute marginals, then develops variational inference through free-energy minimization over tractable exponential families. It concludes with the information-geometric Khan stationary condition, equating the natural parameter to the natural gradient of the expected log-model.

  • Forney-style factor graphs: Forney-style factor graphs represent factorizations with factors as nodes, variables as edges, and incidence determined by factor arguments.The neighboring-edge and neighboring-node notation supports local subgraph descriptions.
  • Belief propagation: Belief propagation uses the distributive law to summarize subgraphs locally, with each factor combining its function and incoming messages before sending an outgoing message.This avoids integrating all variables simultaneously when forming an edge marginal.
  • Variational inference: Variational inference replaces intractable integration with optimization by selecting a tractable candidate density that minimizes reverse Kullback–Leibler divergence to the posterior.The equivalent objective is variational free energy, whose minimum provides an upper bound on −log Z.
  • Exponential families: Restricting the variational family to a regular minimal exponential family makes the optimization finite-dimensional and provides dual natural-parameter and mean-parameter coordinates.The coordinate map is smooth and bijective, with Fisher information as its Jacobian in natural coordinates.
  • Natural gradient and Khan stationary condition: At a variational free-energy minimizer, the natural parameter equals the natural gradient of the expected log-model, which also equals its ordinary gradient in mean coordinates.This is the Khan stationary condition and characterizes the variational fixed point.

3 Variational Message Passing

Variational message passing is formulated as constrained Bethe inference over local beliefs, with the admissible belief space determining the approximation. Different constraints recover BP, structured VMP, or finite edge-family projections from stationarity.

  • Bethe inference and the local polytope: Bethe inference replaces the joint variational density with node and edge beliefs constrained by the local polytope.The resulting problem is a constrained variational optimization over these local beliefs.
  • Known stationary consequences: With the unmodified local polytope, stationarity recovers ordinary sum-product belief propagation and its stationary conditions.BP imposes no finite-dimensional form on node or edge beliefs.
  • Structured VMP: Structured VMP changes the feasible set through node-belief factorization, yielding a hierarchy from BP to mean-field VMP.The coarsest partition recovers BP, the finest gives mean-field VMP, and intermediate partitions produce structured variants.
  • Why constraints matter: In the Normal–Gamma model, exact BP fails to preserve Gaussian and Gamma message families simultaneously, whereas mean-field factorization restores finite closure by averaging out coupling.The omitted uncertainty correction is deferred to Section 5.3.
  • From node factorizations to edge form constraints: Selected edge-family constraints introduce finite edge coordinates whose stationarity yields Fisher-metric projections of local log-messages onto the receiving edge family.Without node factorization, the local log-message is the exact BP log-message; under structured VMP, it is the tilted log-message.

4 Natural-Gradient Message Passing

Natural-gradient message passing localizes Bethe natural-gradient stationarity by constraining edge marginals to exponential families. Each factor sends the receiving edge the Fisher-tangent projection of its exact BP log-message, yielding an edge-local fixed-point scheme whose messages depend on the current receiving marginal.

  • Edge-constrained belief space: Constraining one edge marginal to a regular minimal exponential family replaces its free density with finite natural coordinates while leaving factor beliefs and marginalization multipliers functional.The constrained edge uses qλi, with mean parameter μi and Fisher information Gi(λi).
  • Projected stationarity: At a stationary point, the constrained edge’s natural parameter equals the sum of the projected natural parameters from its two incident factors.The projection retains the component of each exact BP log-message visible in the receiving family’s tangent space.
  • Natural-gradient message passing: NGMP sends the tangent component of an exact BP log-message as a finite natural-parameter message and discards the residual for communication.The residual can remain in functional Lagrange multipliers but is invisible to the finite-dimensional edge marginal.
  • Edge-local extension: The construction extends edge-locally to any constrained subset, while unconstrained edges continue to follow classical BP updates.Each constrained edge receives its own stationarity equation, coupled through the surrounding graph’s exact BP log-messages.
  • Natural-gradient message passing: The projected message depends on the receiving edge’s current marginal, so NGMP repeatedly re-projects each exact BP log-message at that marginal.This makes the update a Fisher-metric fixed-point iteration with receiver-dependent finite messages.

5 Surrogate Models

Surrogate models implement NGMP by replacing non-representable local contributions with conjugate surrogate leaves whose parameters are projected at the receiving edge marginal. This preserves edge-local constrained-Bethe stationarity while enabling ordinary belief propagation on a frozen conjugate surrogate graph.

  • Unary surrogate leaves: Non-conjugate unary leaves are replaced by conjugate pseudo-observations, allowing ordinary belief propagation on an auxiliary graph.The exact local log-message remains the object being projected, while only its representable tangent component is communicated.
  • Unary surrogate leaves: The projected Gaussian message is the mean-coordinate gradient of the exact log-message expectation at the receiving marginal.For Poisson observations, the non-quadratic log-message is replaced by a Gaussian message whose natural parameter is this gradient.
  • Outer fixed-point iteration: The surrogate algorithm alternates ordinary Gaussian BP with recomputation of projected non-conjugate leaves from updated Gaussian edge marginals.The surrogate graph is frozen during the inner BP sweep; the outer loop carries the non-conjugacy.
  • Multi-interface factors: For multi-interface factors, NGMP projects each exact message separately at its own receiving marginal instead of requiring a global joint surrogate family.Each constrained interface keeps its own exponential family and receives a finite, edge-local message.

6 Related Work

Related work positions NGMP relative to VMP, NCVMP, CVI, PVMP, and EP by distinguishing their projection points, information flow, and optimization scope. NGMP recovers NCVMP and PVMP under mean-field constraints while extending the framework to factor-to-edge messages with edge-specific families and receiving marginals.

  • Variational message passing and local constraints: VMP performs local expected-log factor updates, while structured VMP retains selected clusters and related variants preserve this information flow.These methods differ in how local quantities are evaluated, but retain VMP-style information flow.
  • Non-conjugate VMP: NCVMP projects the tilted VMP log-message at the current receiving marginal using its Fisher-metric gradient, but only once per update.Its information flow remains VMP-like because out-of-cluster variables enter through their marginals.
  • From global gradients to factor-to-edge messages: Unlike global CVI, NGMP defines a separate projected message for each factor interface, with its own edge family and receiving marginal.CVI converts a global mean-gradient into an auxiliary conjugate natural parameter, whereas NGMP uses factor-to-edge message semantics.
  • FFG-local CVI marginal optimization: PVMP optimizes a complete edge marginal to convergence, whereas NCVMP takes one natural-gradient step; they share edge stationary points but differ in update cost.Both use VMP messages as input and add a projection at the receiving marginal, making them hybrids in the stated sense.
  • NGMP recovers NCVMP and PVMP under mean-field constraints: Under mean-field or structured-VMP constraints, one NGMP projection step exactly reproduces the NCVMP update, while iterating the edge projection recovers PVMP.The passage states that the NCVMP match has the same numerical result.
  • Expectation propagation and local divergence projections: EP projects a tilted marginal, NCVMP projects a tilted VMP message once, PVMP optimizes a complete edge marginal, and NGMP projects a factor message.The comparison distinguishes these methods by their local projection or optimization object.

7 Comparing VMP to NGMP: When Does NGMP Matter?

NGMP differs from VMP and PVMP by projecting exact cavity log-messages rather than expected log-factors, with benefits emerging when uncertain updates are repeatedly composed. Across smoothing and filtering experiments, NGMP improves uncertainty calibration and resists batching-induced overconfidence, while agreeing with competing methods when edge uncertainty vanishes.

  • Method distinction: VMP uses expected log-factors, whereas NGMP projects exact cavity log-messages onto the receiving marginal family.Both methods keep the graph and selected marginal families fixed; their difference is the information entering each local update.
  • When NGMP matters: When neighboring beliefs concentrate, VMP and NGMP agree; persistent edge uncertainty instead allows discrepancies to accumulate through smoothing or successive filtering updates.The section isolates vanishing uncertainty, latent-chain smoothing, and shared-parameter filtering as the relevant regimes.
  • Poisson smoothing: For 50% sunspot holdout, NGMP propagates cavity uncertainty, with variance growing from 0.09 beside observations to 0.27 in the deepest gaps, while PVMP remains near σ2/2 = 0.05.PVMP’s held-out negative log-likelihood rises from 4.8 at distance one to 33.5 at distances five to eight, whereas NGMP remains near 5.7.
  • Practical rule: Across the ablations, NGMP is a safer default when uncertain updates are repeatedly composed, but its undamped fixed-point iteration can oscillate or fail to converge.The paper therefore damps every NGMP run in natural coordinates.

8 Experiments

Experiments evaluate NGMP in larger heteroscedastic regression and ensemble-forecasting settings. NGMP performs particularly strongly on predictive uncertainty, with gains over competing methods appearing more consistently in NLL than in RMSE.

  • UCI regression: NGMP is evaluated on six UCI regression data sets using a depth-three heteroscedastic hierarchy with 1,000 random Fourier features per feature map.The data sets are Concrete, Energy, Boston Housing, Power Plant, Wine Quality Red, and Yacht; Gaussian weights use closed-form updates while non-conjugate precision links use NGMP.
  • UCI regression: The regression comparison uses 20 deterministic repeated-holdout splits, with 90% of each data set for training and 10% for testing, and paired split indices across methods.Features and targets are standardized using training statistics only.
  • UCI regression: Across the regression benchmarks, NGMP has the lowest mean NLL on Concrete, Energy, Power, and Yacht, while DVI variants are strongest on Boston.Its interval includes the best point estimate on Wine; RMSE is more mixed, with NGMP best on Energy and Power but not improving on DVI for Concrete or Boston.
  • ETTh forecasting: In ETTh forecasting, neural gates often appear competitive on RMSE alone, but their NLL is substantially worse than both precision-gated models.This larger NLL-versus-RMSE separation supports the prediction from the controlled edge-uncertainty experiments.
  • ETTh forecasting: Against PVMP, NGMP has lower NLL at all four ETTh1 horizons and at ETTh2 horizons 192 and 336.NGMP also has lower RMSE at ETTh1 horizons 96, 192, and 720 and at ETTh2 horizons 96, 192, and 336.

9 Discussion and Conclusion

The paper establishes an edge-local natural-gradient stationarity condition in which finite messages are Fisher projections of exact belief-propagation log-messages at receiving marginals. It concludes by proposing compiled surrogate execution and continuous-state active-inference message passing as directions that preserve uncertainty for downstream control.

  • Discussion and Conclusion: NGMP gives fixed-form variational inference an edge-local stationarity condition, with each finite message obtained by Fisher-projecting an exact BP log-message at its receiving marginal.This places BP, structured VMP, EP, and NGMP in one local variational language while retaining different information flows.
  • Compiling factor graphs to surrogate programs: Projected messages on the original factor graph and conjugate leaves on a surrogate graph implement the same projected computation, rather than merely producing agreeing posteriors.The proposed lowering would not change asymptotic arithmetic complexity or the NGMP update.
  • Continuous-state active-inference message passing: AIF-MP is identified as a prospect for continuous-state planning, where exact messages from modified observation and dynamics factors generally do not remain in a tractable finite family.The existing construction uses channel-reparameterized factors but is restricted to discrete state spaces with exact factor evaluations.
  • Discussion and Conclusion: The discussion proposes separating expressive factor-graph modeling from compiled surrogate-program execution, with NGMP translating between them while retaining uncertainty for downstream continuous-state planning.Automating this translation and determining when continuous-state projections are accurate enough for control are presented as central next steps.

A Proof of Theorem 1

The proof derives Theorem 1 by separately varying factor and edge terms, then closing the multipliers through marginalization constraints. Exponential-family identities reduce the resulting projected stationarity condition to equation 37.

  • Factor-side variation: The factor-side variation leaves adjacent factor beliefs unconstrained and therefore follows the corresponding Bethe-Lagrangian variation.The edge-form constraint changes the edge variable but does not constrain adjacent factor beliefs.
  • Edge-side variation: The edge-side variation collects all λ_i-dependent terms and applies exponential-family identities to recover the edge stationarity condition.Multiplication by G_i(λ_i)^−1 yields its projected form.
  • Closing the multipliers: Marginalization constraints close the multipliers by substituting the preceding identity into the constraint and rewriting it in log-coordinates.The proof uses the definition in equation 35 to obtain the corresponding log-coordinate identity.
  • Closing the multipliers: Covariance with T_i removes constants, while the affine form of log q_{λ_i} supplies the identity needed for the final substitution.The proof notes that log q_{λ_i} is affine in T_i, so its relevant derivative equals λ_i.
  • Conclusion: Substituting both identities into the projected stationarity condition produces equation 37.The proof explicitly identifies the resulting expression as equation 37.

B Supplementary Derivation for Information-Geometric Message Passing · B.1 Exact Constrained Model

The appendix uses a deliberately small one-latent-variable example to clarify constrained exponential-family messages. Its exact derivation shows that functional multipliers enforce marginal constraints, while finite messages retain only representable projected components and cannot transmit arbitrary non-conjugate residuals.

  • B Supplementary Derivation for Information-Geometric Message Passing: The appendix presents a pedagogical inductive-inference example rather than a new algorithmic contribution.It uses one latent variable and two unary factors, avoiding hidden factorization approximations in the notation.
  • B Supplementary Derivation for Information-Geometric Message Passing: The derivation clarifies that constrained free-energy multipliers are functional objects, whereas algorithmic messages are finite exponential-family surrogates.The surrogates can be computed from the functional multipliers without making a non-conjugate factor conjugate.
  • B Supplementary Derivation for Information-Geometric Message Passing: The construction follows constrained variational logic: a positive unnormalized density, a free-energy objective, admissibility constraints, and stationary conditions from variational calculus.The example follows the constrained free-energy logic of Caticha and the constrained Bethe-free-energy construction of Senöz et al.
  • B.1 Exact Constrained Model: In the exact constrained model, z connects to two unary factors, one conjugate and one non-conjugate, while its marginal qλ is constrained to the chosen exponential family E.The two factor beliefs remain unrestricted, and densities use the carrier reference measure dν(z) = h(z) dz.
  • B.1 Exact Constrained Model: Functional multipliers ρc and ρn enforce exact local marginalization constraints in the constrained Bethe formulation.The Lagrangian includes the constraints, and variational differentiation with respect to factor beliefs yields the corresponding stationary relations.
  • B.1 Exact Constrained Model: Exact marginalization is not the obstruction: functional multipliers can absorb the required terms to make factor beliefs equal qλ.When a factor is non-conjugate, the exact multiplier generally contains a negative non-conjugate residual of log fa and is therefore not an exponential-family message.
  • B.1 Exact Constrained Model: Stationarity with respect to λ identifies only multiplier components visible through the sufficient statistics T, leaving orthogonal functional components free to enforce exact constraints.The resulting condition is a stationary condition on qλ, not the density equality qλ(z) ∝ fc(z)fn(z).
  • B.1 Exact Constrained Model: Each mean-coordinate gradient gives the natural parameter of an outgoing message, but a finite algorithm cannot communicate an arbitrary non-conjugate residual.The exact constrained problem remains well-defined because unrestricted factor beliefs can cancel that residual through the functional multiplier.

B.2 Projected Surrogate Update · B.3 Computing the Tangent Projection in Practice

The projected surrogate replaces the local graph with a conjugate, finite-dimensional update whose fixed points coincide with stationary points of the original constrained problem. In practice, tangent projections use analytic expectations, unscented approximations, or quadrature depending on the factor and marginal family.

  • B.2 Projected Surrogate Update: The surrogate update preserves the original graph topology while replacing local factors with conjugate sites in the chosen sufficient statistics.Densities use the carrier measure dν(z) = h(z) dz; Lebesgue densities are recovered by multiplying by h(z).
  • B.2 Projected Surrogate Update: For conjugate factors, projection recovers the usual natural-parameter contribution; for non-conjugate factors, it retains only the component visible under the current marginal.The projection is defined relative to the receiving marginal qλ0.
  • B.2 Projected Surrogate Update: The inner surrogate problem freezes λ0, solves the constrained Bethe problem, and defines an update map rather than enforcing λ0 = λ during optimization.Differentiating through the frozen projected contribution would introduce extra ∂ηa/∂λ0 terms.
  • B.2 Projected Surrogate Update: The outer iteration λt+1 = Φ(λt) is a fixed-point method, equivalently a unit natural-gradient step, and any converged fixed point is stationary for the original constrained problem.This does not establish exactness of intermediate surrogate marginals or global optimality for non-convex reduced objectives.
  • B.2 Projected Surrogate Update: At a fixed point, half-edge multipliers become finite-dimensional exponential-family messages, while the original constrained problem’s exact multipliers remain functional objects.The finite message represents the conjugate surrogate factor, not a transformation of the original non-conjugate factor into a conjugate one.
  • B.3 Computing the Tangent Projection in Practice: The tangent projection reduces to covariances between sufficient statistics and the log-message under the current marginal.Whether it is closed form depends on analytic availability of the required expectations.
  • B.3 Computing the Tangent Projection in Practice: In the Poisson state-space model, Gaussian expectations are analytic, so NGMP performs a single closed-form natural-gradient step per edge.PVMP iterates the same closed-form gradients inside its inner manifold optimizer, making both methods closed-form in this model.
  • B.3 Computing the Tangent Projection in Practice: When expectations are unavailable analytically, NGMP uses deterministic unscented approximation for the normal mean–precision model and 32-node quadrature for the sequential heteroskedastic model.These numerical variants depend on the selected sigma-point or quadrature rule rather than sampling noise.

C Damping and Momentum · C.1 Vector-transport momentum

The surrogate fixed-point map supports damping and momentum in natural/message coordinates, preserving the stationary equations while changing convergence paths. Vector-transport momentum adapts heavy-ball updates to the exponential-family geometry and enables convergence where damping alone stalls.

  • C Damping and Momentum: The outer map Φ(λ) builds projected surrogate leaves at current constrained marginals, freezes the surrogate graph, and runs a conjugate belief-propagation sweep.In mean coordinates, the reduced constrained-Bethe objective has the same stationarity form as the global exponential-family free energy.
  • C Damping and Momentum: Damping updates natural parameters as λ(t+1) = (1 −α)λ(t) + α Φ(λ(t)), with 0 < α ≤1, changing only the step length.The undamped update is a unit natural-gradient or fixed-point step.
  • C Damping and Momentum: For individual surrogate messages, damping averages projected natural parameters before converting them back to message-function parameters.For the Gaussian Poisson leaf, canonical coordinates (ξ, τ) are averaged before conversion to pseudo-observation parameters (ey, Λ).
  • C Damping and Momentum: On sunspot masks, α = 1, α = 0.25, and heavy-ball α = 0.5, β = 0.2 reach the same fixed point by iterations 5, 14, and 8, respectively.Damping costs iterations when counts are moderate and initialization log(yk + 1) is near the fixed point.
  • C Damping and Momentum: Extreme latent log-rate excursions can stop the fixed-point map from contracting, motivating damping for Poisson state-space trajectories with strongly reactive natural-gradient messages.Figure 14 evaluates simulated series with σ2 = 0.1, all counts observed, lengths N = 100, 250, 500, and 1000, using 20 seeds and 200 variational iterations.
  • C Damping and Momentum: Momentum applies to the natural-parameter residual Φ(λ) −λ = −∇µFexact(λ), and converged residuals remain zero, so acceleration changes the path but not stationary equations.Other accelerated fixed-point variants differ in where Φ is evaluated, but retain the same coordinate principle.
  • C.1 Vector-transport momentum: Vector-transport momentum carries the previous tangent vector to the current iterate on the Fisher-metric exponential-family manifold before combining it with the new residual.The transport rescales each element by the square root of the diagonal-Fisher-metric ratio and caps the resulting step’s Euclidean norm; β > 0 can leave the natural domain.
  • C.1 Vector-transport momentum: In depth-three hierarchies, damping alone stalls within the sweep budget, whereas vector-transport momentum converges.This motivation follows the reported benefit of momentum for natural-gradient Gaussian variational approximations.

D Convergence Diagnostics and Additional Results for the Comparison Study · E Experiment details and additional results · E.1 Regression details

The appendix documents convergence behavior, numerical failure modes, and budget-matched controls for NGMP and PVMP, then specifies the regression models, uncertainty representations, and evaluation setup. Across the comparison studies, both methods generally plateau within their sweep budgets, while NGMP’s predictive behavior is visually stable across batch-processing schemes.

  • D Convergence Diagnostics and Additional Results for the Comparison Study: Both methods reach their Poisson plateaus within 20 sweeps, so the 50% holdout separation reflects fixed points rather than premature stopping.NGMP plateaus by sweep three to four, while PVMP plateaus shortly afterward.
  • D Convergence Diagnostics and Additional Results for the Comparison Study: A default gradient-norm bound can trap tail marginals, producing mask-average held-out negative log-likelihood up to 66 from a single month with rate ≈104.The projection returns its input, and neither damping nor additional sweeps escapes the failure.
  • D Convergence Diagnostics and Additional Results for the Comparison Study: Posterior predictive bands and predictive-variance decompositions are visually indistinguishable across the two full-batch fits and NGMP’s sequential fit; only PVMP’s sequential fit differs.The figures use identical axes and separate epistemic and aleatoric predictive variance components.
  • D Convergence Diagnostics and Additional Results for the Comparison Study: Budget-matched NCVMP limits each ProjectedTo call to one inner iteration, matching PVMP’s graph, constraints, initialization, and outer sweep schedule.NGMP uses one natural-gradient step per edge update, while PVMP can use up to 100 gradient evaluations per update.
  • E.1 Regression details: We use repeated 90/10 train–test splits, with feature counts defined before standardization and random-feature construction and capacity separated into predictive parameters P and uncertainty scalars U.The regression comparison reports full-test NLL and RMSE in original target units over 20 paired splits.
  • E.1 Regression details: Each regression feature map uses 1,000 random Fourier features for a Matérn-3/2 kernel, concatenated with standardized inputs and an intercept across three spectral scales.The feature maps are sampled once per split and then held fixed; Gaussian weight priors are used at all layers.
  • E.1 Regression details: Regression fits use at most 60 sweeps, a relative weight-mean stopping threshold of 10−5, diagonal jitter 10−8, and momentum parameters α = 0.6 and β = 0.8.Baseline variants include homoscedastic, heteroscedastic, and learned-output-covariance configurations, with hoIVON evaluated over K = 20 posterior network draws.

E.2 Ensemble forecasting details

The section details univariate long-horizon ETTh1/ETTh2 forecasting with a five-expert frozen neural bank and an equally weighted 1,000-component Gaussian mixture. It specifies mixture-based NLL and point-prediction evaluation and provides full results in Table 11.

  • Data and forecasting setup: Forecasting uses 96 hours of multivariate context to predict standardized oil temperature 96, 192, 336, or 720 hours ahead on hourly ETTh1 and ETTh2 data.The frozen expert bank contains five independently trained neural forecasters.
  • Ensemble construction: Each component combines frozen-expert predictions using softmax logits from one draw, producing a Gaussian mean and variance.The supplied passage defines the mean as a softmax-weighted sum of expert predictions and references the corresponding variance formula.
  • Evaluation and reporting: The reported posterior predictive is an equally weighted mixture of 1,000 components; NLL uses log-sum-exp, while point prediction averages component means.Table 11 includes full ETTh results, including Adam-trained gates omitted from the main table.
Loading 2608.15922v1…