PaperlayerView exact revision on arXiv

Source-linked AI summary

Direct Message Approximation (DMA): A Consistency-Based Framework for Tractable Approximate Inference on Factor Graphs

Ralf Herbrich, Rainer Schlosser, Jan Lemcke, Johann Ukrow, Anna Kazachkova, Nicolas Alder, Leonhard Hennicke, Theo Bardey, Nico Grimm, Luca Kleinschmidt, Philipp Kolbe, Cezary Kujath, Johanna Schlimme, Karl Matti Schütz

arXiv:2609.29466v1cs.LGcs.AIstat.ML

TL;DR

EP and VMP approximate edge marginals, leading to iterative schedules, possible negative-precision messages, and VMP point-mass collapse at deterministic factors. DMA directly approximates factor-to-variable messages and supplies consistency and error guarantees, including a separate bound for the improper product-backward message. Its BNN instantiation reports widening extrapolation uncertainty, including under model mismatch.

  • Problem

    EP and VMP recover factor-to-variable messages from approximate edge marginals, a design associated with iterative updates, negative-precision messages, and VMP collapse at deterministic factors.

    Source: §1 · p. 2

  • Method

    DMA approximates factor-to-variable messages directly, defines consistency at concentrated inputs, and derives explicit messages for product and leaky-ReLU factors.

    Source: §1 · pp. 2–3, §2.1 · p. 3, §3 · p. 5

  • Results

    DMA's BNN inference produces extrapolation uncertainty that widens in data-sparse regions and useful predictive intervals under model mismatch.

    Source: §6 · p. 10

  • Takeaways & Limitations

    The BNN instantiation requires no gradient learning-rate hyperparameter and carries the reported structural guarantees into larger models without sacrificing epistemic quality for scalability.

    Source: §6 · p. 10

    The master theorem requires proper outgoing messages, and DMA's fully factorised Gaussian belief does not directly provide non-diagonal block structure without additional moment formulas.

    Source: §6 · p. 10

Abstract

from arXiv · show

Approximate message passing on factor graphs underlies two dominant families of probabilistic inference algorithms: expectation propagation (EP) and variational message passing (VMP). Both methods approximate the marginal at each factor edge, forcing an iterative round-robin schedule, risking negative-precision messages, and, for VMP, collapsing to point estimates at Dirac-delta factors. We introduce Direct Message Approximation (DMA), which approximates factor-to-variable messages directly rather than the marginal. For normalisable factors, we define a consistency condition (requiring exactness when all other incoming messages are Dirac deltas) to guide message construction. We prove a master theorem (proper messages, any graph) bounding marginal KL from message KL, with three structural corollaries: Dirac-input consistency, no EP-style inner-loop iteration, and no negative-precision messages. Further, we prove a complementary $O(1/r^2)$ guarantee for the inherently improper backward message of the product factor, whose closed-form treatment has resisted prior work. As a concrete instantiation, we derive explicit DMA messages for the product and leaky-ReLU factors and assemble a Bayesian neural network (BNN) inference algorithm with one forward/backward sweep per training example and no gradient learning-rate hyperparameter, validating that the structural guarantees translate to predictive uncertainty that widens in data-sparse regions, including under model mismatch.

1 INTRODUCTION

EP and VMP approximate edge marginals and recover messages by division, creating iterative updates, possible invalid Gaussian messages, and Dirac collapse in VMP. DMA instead approximates factor-to-variable messages directly, using consistency with exact messages at concentrated inputs as its design criterion.

1 INTRODUCTION

  • Marginal-based updates require round-robin sweeps, can yield negative-precision Gaussian EP messages, and collapse deterministic-factor updates to point masses in VMP.The VMP collapse prevents a non-trivial weight posterior in Bayesian neural networks.Source: §1 · p. 2
  • DMA constructs factor-to-variable approximations directly, avoiding the cavity division that links outgoing messages to incoming messages.Its consistency condition requires exact-message recovery when all other inputs are point masses.Source: §1 · p. 2
  • DMA differs from ADF by adding a formal consistency axiom, an edge-local master theorem, and a separate guarantee for the improper product-backward message.The stated master bound requires no graph contraction condition.Source: §1 · p. 2
  • The master theorem bounds marginal KL by message KL edge-locally on any graph, while the product-backward message has a separate O(1/r^2) guarantee.The latter addresses an inherently improper message outside the theorem's proper-message scope.Source: §1 · pp. 2–3

2 DIRECT MESSAGE APPROXIMATION

DMA approximates outgoing messages without forming edge marginals, and its local theory connects message quality to marginal error. A consistency condition, moment-matching construction, and structural corollaries support the framework's use in BNN inference.

2.1 DEFINITION AND CONSISTENCY CONDITION

  • DMA requires a message approximation to converge to the exact message as the other incoming messages concentrate at point values.The consistency condition applies to normalisable factor slices; the product-backward message is handled separately because it is improper for general Gaussian inputs.Source: §2.1 · pp. 3–4
  • The consistency condition is necessary but not sufficient: moment matching selects a specific approximation, while the master theorem bounds its marginal error at any input width.The section frames these as local guarantees for individual message computations.Source: §2.1 · p. 4

2.2 CONSTRUCTION RECIPE

  • For deterministic factors, the construction recipe computes the first two moments of the function output under independent Gaussian inputs and projects them onto a Gaussian message.In the Dirac limit, the message converges to the exact point mass at the function value.Source: §2.2 · p. 4

2.3 MASTER THEOREM AND CONSEQUENCES

  • The master theorem bounds marginal error using the quality of a single outgoing message and the incoming message at that edge, without a global contraction condition.Across 192 leaky-ReLU configurations, the bound held in every case, with maximum normalised ratio 0.97.Source: §2.3 · p. 4
  • DMA messages need no EP-style inner-loop fixed-point iteration and cannot have negative precision; piecewise-C2 factors also retain vanishing error in the concentrated-input limit.The BNN algorithm uses one forward sweep and one backward sweep per training example, while training still repeats sweeps across examples and epochs.Source: §2.3 · p. 5

3 DMA FOR NON-GAUSSIAN FACTORS

The section derives Gaussian DMA messages for the product and leaky-ReLU factors used in feedforward BNNs. The product-backward message is improper and receives a separate concentrated-input guarantee; the leaky-ReLU messages use truncated-Gaussian moment matching.

3.1 THE PRODUCT FACTOR

  • The product factor has exact forward moments under independent Gaussian inputs, followed by Gaussian projection, while its backward message is inherently improper.The backward message uses a log-normal intermediate because the Gaussian ratio has no finite mean.Source: §3 · p. 5, §3.1 · p. 5

3.2 APPROXIMATION GUARANTEE FOR THE PRODUCT BACKWARD MESSAGE

  • The product-backward DMA has a C/r^2 error bound against a proper truncated reference when both input signal-to-noise ratios are at least 2 and their ratio is bounded.The exact backward message is improper, so it falls outside the master theorem's proper-message requirement.Source: §3.2 · p. 6

3.3 THE RELU FACTOR

  • Leaky-ReLU forward messages use moment matching for a mixture of positive and scaled negative truncated-Gaussian pieces.The experiments focus on α > 0; the standard ReLU backward message is improper.Source: §3.3 · pp. 6–7
  • For α > 0, the leaky-ReLU backward message is normalized using a correction determined by α before matching moments to a proper Gaussian.The piecewise Jacobian contributes a factor 1/α on the x ≤ 0 branch.Source: §3.3 · p. 7

4 APPLICATION: BAYESIAN NEURAL NETWORK INFERENCE

The paper builds a feedforward BNN factor graph from Gaussian priors, product factors, sum factors, and leaky-ReLU factors, then applies DMA in one forward and one backward sweep per example. Its regression experiments report widening predictive uncertainty outside the training region and competitive results on a larger network.

4 APPLICATION: BAYESIAN NEURAL NETWORK INFERENCE

  • The application assembles the product and leaky-ReLU DMA factors into an inference algorithm for feedforward BNNs composed of Gaussian, product, sum, and leaky-ReLU factors.It evaluates the algorithm on a scalar regression task.Source: §4 · p. 7

4.1 FACTOR GRAPH OF A FEEDFORWARD NETWORK

  • The BNN factor graph uses Gaussian prior factors, product factors for matrix-vector products, sum factors, and leaky-ReLU factors; DMA supplies messages for the product and activation factors.ReLU factors are the dominant source of nonzero message error δ when inputs are uncertain.Source: §4.1 · p. 7

4.2 INFERENCE ALGORITHM: ONE FORWARD/BACKWARD SWEEP PER EXAMPLE

  • Each training example uses one forward sweep followed by one backward sweep, with each factor-to-variable message computed once and no EP-style inner-loop iteration.The algorithm runs in O(P l d_l d_{l−1}) time per example, with a reported 1.8× per-epoch overhead over a standard neural-network pass.Source: §4.2 · pp. 7–8

4.3 ILLUSTRATIVE EXPERIMENT: 1D REGRESSION

  • The posterior predictive mean tracked the true function within [−2.5, 1.5], and its ±2σ intervals widened in extrapolation regions without training data.Figure 2 also displays posterior weight beliefs after training.Source: §4.3 · p. 8, Figure 2 · p. 8
  • On the correctly specified 1D task, DMA’s median extrapolation NLL was 0.80 versus 4.05 for AdamW, while both had similar calibration error over 20 seeds.DMA used no gradient learning-rate hyperparameter; under model mismatch, its predictive distribution still widened outside the training range while Adam provided no epistemic uncertainty.Source: §4.3 · p. 9
  • DMA’s 1,932-weight network achieved competitive results in 3 epochs (0.72 s total), versus 1.7 s for Adam, which required ~100 epochs.The network had four outputs and architecture 6→6→12→48→24→4.Source: §4.3 · p. 9

5 RELATED WORK

The related work contrasts DMA with approximate-inference methods that target marginals or avoid backward messages. It distinguishes DMA through direct message construction, backward weight updates, and its consistency and error-bound guarantees.

5 RELATED WORK

  • DMA’s master theorem differs from Ihler et al. (2005) by bounding projected-message errors edge-locally without a graph contraction condition.TrueSkill demonstrates EP at scale on conjugate factors, while Infer.NET provides pluggable message operators for EP/VMP.Source: §5 · p. 9
  • EP variants retain cavity division and its pathologies, while DMA directly approximates outgoing messages; for BNNs, this eliminates per-example cavity computation and iterative sweeps.The cited BNN EP approach is Hernández-Lobato & Adams (2015).Source: §5 · p. 9
  • ADF also avoids cavity division, but its forward-only architecture does not propagate output-likelihood information backward to update upstream weight beliefs.DMA’s backward product message is designed to handle that step.Source: §5 · p. 9
  • VMP and ELBO-based methods use marginal-first division, which collapses deterministic-factor posteriors to Dirac deltas; Bayes by Backprop, MC Dropout, and SWAG do not maintain explicit factor-graph message structure.Source: §5 · p. 9
  • Diagonal Laplace fits a Gaussian post hoc at the MAP, while IVON propagates uncertainty through gradient statistics; neither provides a backward product message to weight beliefs.Source: §5 · p. 9

6 DISCUSSION AND CONCLUSIONS

The discussion presents DMA’s theoretical guarantees and BNN results, while identifying assumptions and open questions that constrain those conclusions. It also summarizes the paper’s supporting materials and background.

6 DISCUSSION AND CONCLUSIONS

  • For proper messages on any graph, the master theorem bounds marginal KL by O(δ) with a topology-independent, edge-local prefactor; the corollaries establish no EP-style inner-loop iteration, no negative-precision messages, and Dirac-input consistency at linear factors.Source: §6 · pp. 9–10
  • For the improper product-backward message, Theorem 3.3 gives a C/r^2 bound in the concentrated-input regime, and Appendix C.3 empirically confirms the rate against importance-sampling reference marginals.Source: §6 · p. 10
  • The BNN results report widening extrapolation uncertainty under model mismatch, no gradient learning-rate hyperparameter, and scalability to substantially larger networks at wall-clock time comparable to Adam.The paper reports calibration comparable to diagonal Laplace on smaller models.Source: §6 · p. 10
  • The master theorem requires proper outgoing messages, and DMA’s fully factorised Gaussian belief leaves non-diagonal-input moment formulas necessary for block-diagonal extensions.Source: §6 · p. 10
  • Bounding accumulated error after K > 1 sweeps on cyclic graphs is identified as the most pressing open problem for end-to-end convergence guarantees.The construction recipe can extend to factors with closed-form moment formulas; Student-t and Gaussian-mixture messages are proposed as targets.Source: §6 · p. 10

AI USE STATEMENT

REPRODUCIBILITY STATEMENT

A.2 THE PROJECTION STEP AND THE α-DIVERGENCE FAMILY

  • The appendix overview includes factor-graph and sum-product background, while the α-divergence discussion connects its limiting cases to forward KL for EP and reverse KL for VMP.Source: APPENDIX OVERVIEW · p. 13, §A.2 · p. 14

A.3 PRIOR APPROXIMATE INFERENCE METHODS

This appendix reviews EP and VMP’s marginal-first message construction and ADF’s forward-only alternative, then provides proof material on moment matching and the DMA master theorem.

A.3 PRIOR APPROXIMATE INFERENCE METHODS

  • EP projects the true marginal onto the approximation family and divides out the incoming message; when the marginal is outside that family, the approximation introduces local error and must iterate to a fixed point.VMP also recovers messages by dividing out the incoming message, while standard ADF avoids division but has no backward sweep for upstream weight updates.Source: §A.3 · p. 15

B.1 PROOF OF THEOREM A.1 (MOMENT-MATCHING MINIMISER OF KL)

  • For an exponential family, the KL minimiser matches expected sufficient statistics; for Gaussian Q, this reduces to matching the first two moments.Source: §B.1 · p. 16

B.2 PROOF OF THEOREM 2.3 (DMA MASTER THEOREM)

  • The master-theorem proof uses a nonnegative Bregman-divergence decomposition and bounds the incoming message by its supremum norm to obtain the stated error bound.The proof passage assumes the incoming message is bounded and uses the message KL error δ.Source: §B.2 · p. 16

B.3 PROOFS OF COROLLARIES 2.4, 2.5, AND 2.6

The corollary proofs show that direct message construction avoids message recomputation and negative precision, while recovering exact messages in key limiting cases. For the product factor, a log-normal intermediate yields a backward-message approximation that is consistent as the denominator input concentrates.

B.3 PROOFS OF COROLLARIES 2.4, 2.5, AND 2.6

  • Because each outgoing message uses only neighboring incoming messages, updating it leaves the same-edge incoming message unchanged and triggers no neighboring-message recomputation.Source: §B.3 · p. 17
  • Moment matching gives Gaussian messages with nonnegative precision, tending to the correct Dirac delta as input variance vanishes.Conjugate Gaussian factors are exact, and product and leaky-ReLU forward messages have vanishing marginal KL in the concentrated-input limit.Source: §B.3 · p. 17

B.4 PROOFS OF PROPOSITIONS 3.1 AND 3.2 (PRODUCT FACTOR DMA)

  • The product backward ratio has no finite moments under Gaussian inputs, so its approximation uses a log-normal intermediate to derive the ratio moments.The direct Gaussian ratio moments diverge because of the 1/|y| singularity at zero.Source: §B.4 · p. 17
  • As the denominator input concentrates, the product backward approximation’s mean and variance converge to those of the exact inverse constraint.The cited verification gives ˆτx/ˆρx → µz/ȳ and 1/ˆρx → σ²z/ȳ².Source: §B.4 · p. 18

B.5 PROOF OF THEOREM 3.3 (CONCENTRATED-INPUT BOUND FOR PRODUCT BACKWARD MESSAGE)

The proof bounds the product backward-message approximation in the concentrated-input regime despite the exact message being improper. The appendix also states the leaky-ReLU boundary at standard ReLU, maps theorem coverage across factor directions, and describes six empirical validation checks.

B.5 PROOF OF THEOREM 3.3 (CONCENTRATED-INPUT BOUND FOR PRODUCT BACKWARD MESSAGE)

  • The exact product backward message is improper, so the proof truncates the ratio’s non-normalisable tail and controls the approximation through non-Gaussianity and moment error.The two KL terms are bounded using the chain rule and data processing for non-Gaussianity, and a Taylor expansion for moment error.Source: §B.5 · p. 18
  • The product backward-message KL is O(1/r²) in the concentrated-input regime, with the signal-to-noise ratio ratio bounded as specified.The proof decomposes the KL into two terms, each bounded by Cκ/r².Source: §B.5 · pp. 18, 20

B.6 PROOFS OF PROPOSITIONS 3.4 AND 3.5 (RELU FACTOR DMA)

  • For leaky-ReLU, forward and backward DMA recover the exact inverse or activation in the zero-variance limit; at α=0 the backward message is improper and formulas (11)–(12) do not apply.The paper uses a small positive slope or a uniform prior on the negative half-line in practice; standard ReLU is not used, and its analogous concentrated-input bound is left for future work.Source: §B.6 · pp. 22–23, §B.7 · p. 23

B.7 THEOREM COVERAGE BY FACTOR AND DIRECTION

  • The theorem-coverage map assigns proper messages to Theorem 2.3 and improper messages to Theorem 3.3; linear and copy factors satisfy Corollary 2.4 exactly, while Gaussian likelihoods are conjugate and exact.Source: §B.7 · p. 23, Table 2 · p. 23

C EMPIRICAL VALIDATION OF DMA APPROXIMATION QUALITY

  • Six empirical checks cover the master bound, its tightness, factor-level SNR rates, and DMA-versus-reference marginal comparisons for product and leaky-ReLU factors.The described checks include 192 leaky-ReLU configurations and importance-sampling reference marginals; the passage reports the validation plan and stated confirmations, not benchmark comparisons.Source: C EMPIRICAL VALIDATION OF DMA… · p. 24

C.1 MASTER THEOREM VALIDATION

This section validates DMA’s marginal-error bound and examines its behavior across parameter sweeps and factor types, while describing the BNN inference procedure. It compares DMA with Adam and AdamW on extrapolation and training behavior, noting that DMA outperformed AdamW on extrapolation NLL without learning-rate or weight-decay search, while AdamW provided no uncertainty quantification and standard Gaussian EP was not a viable baseline.

C.1 MASTER THEOREM VALIDATION

  • Across 192 leaky-ReLU configurations, every marginal-KL-to-bound ratio is below 1, with a maximum of 0.97.The message and marginal errors were computed analytically on a fine grid, without Monte Carlo; the true backward message is proper for α > 0.Source: Figure 3 · p. 24, §C.1 · p. 24

C.2 BOUND TIGHTNESS: PER-PARAMETER SWEEPS

  • Across parameter sweeps, the bound exceeds actual marginal KL; the maximum ratio is 0.68, while the actual KL peaks near r ≈ 2–3.As α approaches 1, both errors decrease toward the linear-factor case; as σx widens, actual KL rises while the bound falls, making it most conservative at small σx.Source: §C.2 · p. 25, Figure 4 · p. 25

C.4 COPY FACTOR: GAUSSIAN–LOG-NORMAL APPROXIMATION

  • The product backward-message approximation uses two moment-matched copy-factor steps, each with O(1/r^2) KL error.The first bridges log-normal to Gaussian and the second Gaussian to log-normal; the first-step KL depends on log-space variance, not its mean parameter.Source: §C.4 · pp. 25–26

C.3 SNR SWEEP: FACTOR-LEVEL APPROXIMATION QUALITY

  • For product and leaky-ReLU factors, the estimated marginal KL decays as O(1/r^2) with input SNR.The sweep estimates true marginals using 500,000 samples per factor and SNR value, then compares them with DMA marginals.Source: §C.3 · p. 25, Figure 5 · p. 26

C.5 PRODUCT FACTOR

  • For the product factor’s improper backward message, the marginal KL follows the predicted O(1/r^2) rate across two decades of r.The KL is already small at r = 2 and several orders of magnitude smaller at r = 10; under stress, backward marginals become severely non-Gaussian while the forward Z marginal remains well approximated.Source: §C.5 · pp. 26–27, Figure 5 · p. 26, Figure 7 · p. 27

C.6 RELU FACTOR

  • For α = 0.1, the proper leaky-ReLU backward-message KL decays as O(1/r^2), consistent with the master theorem’s message-KL bound.Unlike the product factor, the backward message is proper for α > 0; standard ReLU at α = 0 is improper. DMA captures the bulk location but not the non-Gaussian backward shape or forward point mass at y = 0.Source: §C.6 · p. 27, Figure 8 · p. 27

D INFERENCE ALGORITHM

  • The BNN procedure processes mini-batches with stored outgoing weight messages and a forward/backward sweep, without per-example cavity states.It divides out each batch’s previous message before processing that batch, and checks convergence once per epoch using normalized average log-likelihood.Source: D INFERENCE ALGORITHM · pp. 28, 30

E.2 COMPARISON WITH ADAM

  • DMA reaches extrapolation NLL 0.54 versus Adam’s 6.6 nats/example, stopping at epoch 17 versus Adam’s roughly 80 epochs.DMA’s predictive variance grows outside the training region; Adam’s point estimate provides no uncertainty quantification. DMA’s per-epoch cost is 1.8× Adam’s, but its earlier stopping yields about 5× lower total training time.Source: §E.2 · p. 31, Figure 9 · p. 31

E.3 COMPARISON WITH ADAMW

  • DMA achieved median extrapolation NLL 0.80 (IQR 1.95), versus the best AdamW result of median 4.05 (IQR 10.81) at η = 0.01, λ = 1.0.The comparison covers 20 independently drawn datasets and true functions on the 83-weight 1D regression task; Table 3 reports median [IQR].Source: §E.3 · p. 32, Table 3 · p. 33
  • AdamW (Loshchilov & Hutter, 2019) remained a point estimate with a fixed ±2β predictive band, while DMA required no learning-rate or weight-decay search.The tested AdamW settings also showed hyperparameter sensitivity: η = 0.01 converged slowly, η = 1.0 diverged at λ = 1.0, and λ = 0.01 failed at η = 0.1.Source: §E.3 · p. 32, Figure 10 · p. 32

E.4 WHY EP IS NOT A VIABLE BASELINE

  • Standard Gaussian EP (Minka, 2001) was not reported as a single baseline because stabilisation heuristics require implementation-specific choices and tuning.Moment matching followed by cavity division can yield negative-precision sites when the projected marginal is wider than the cavity, a failure associated with the network’s nonlinear product and ReLU factors.Source: §E.4 · pp. 32–33
  • DMA avoids EP’s negative-precision site failure by constructing outgoing Gaussian messages directly from the joint factor, without cavity division.The comparison refers to the theoretical analysis in Table 1 (Appendix A) and Section A.3.Source: §E.4 · p. 33

E.5 COMPARISON WITH IVON

On the 83-weight regression task, DMA converges without learning-rate tuning and is more reliable across seeds than IVON; diagonal Laplace is comparable on small-scale extrapolation metrics but has a Hessian-based scaling cost. The calibration section compares the methods, while the following model-mismatch setup uses a narrower inference network than the data-generating network.

E.5 COMPARISON WITH IVON

  • DMA converges at epoch 17 without a learning rate, whereas IVON’s selected learning rates either remain unconverged, converge slowly, or oscillate within 500 epochs.At η=0.01, IVON converges to NLL ≈0.26 with δ=0.5.Source: §E.5 · p. 33, Figure 12 · p. 34
  • IVON diverges on 14 of 20 seeds after an exhaustive hyperparameter sweep, while DMA remains finite on every seed and has lower median extrapolation NLL (0.79 vs. 2.11).The finite-seed IVON median has IQR 5.27, versus DMA’s 1.28.Source: §E.5 · p. 34

E.6 COMPARISON WITH THE DIAGONAL LAPLACE APPROXIMATION

  • On one representative seed, diagonal Laplace achieves extrapolation NLL 0.37 nats/example and DMA achieves 0.54, while both track the true function in the training region.The paper identifies computational cost as their key differentiator.Source: §E.6 · p. 35, Figure 14 · p. 36

E.7 CALIBRATION

  • With matched He priors, DMA and diagonal Laplace are broadly comparable over 20 seeds in extrapolation NLL and calibration error.Median calibration errors are −0.09 and −0.10, respectively; median extrapolation NLLs are 0.82 and 0.77.Source: §E.7 · p. 36, Table 4 · p. 37
  • Diagonal Laplace’s exact Hessian costs O(NP^2) time and O(P^2) memory, limiting its practicality as the parameter count grows.The paper contrasts this with DMA’s cost of O(NP · sweeps), linear in P.Source: §E.7 · pp. 36–37

F.1 SETUP

  • The model-mismatch experiment compares DMA and Adam using the same two-hidden-layer inference network, narrower than the data-generating network.The data-generating network has widths 12 and 10; the inference network has widths 6 and 5.Source: §F.1 · p. 37

F.2 RESULTS

Under model mismatch, DMA provides predictive intervals that widen outside the training range, unlike the fixed bands from point-estimate Adam and AdamW. AdamW can achieve a better median extrapolation NLL through weight decay, while DMA has the tightest IQR among the three methods.

F.2 RESULTS

  • DMA’s median extrapolation NLL is 0.01 (IQR 0.44), versus Adam’s 0.50 (IQR 2.23); AdamW reaches −0.36 (IQR 0.73), while only DMA provides widening uncertainty intervals.AdamW’s weight decay is described as implicit regularisation; all comparisons are over 20 seeds.Source: §F.2 · p. 37, Table 5 · p. 38, Figure 16 · p. 38
  • Adam and AdamW use point predictions with fixed ±2β bands, whereas DMA’s predictive intervals widen outside the training range under model mismatch.The discussion states that the fixed bands convey no information about epistemic uncertainty.Source: §F.2 · p. 38, Figure 16 · p. 38

G LARGER-SCALE BNN EXPERIMENT

The appendix presents a larger-scale BNN experiment, demonstrating that DMA scales to a substantially larger network without algorithmic changes. At 1,932 weights, the available Bayesian baseline comparison is constrained by the computational demands and limitations of the alternatives.

G LARGER-SCALE BNN EXPERIMENT

  • At 1,932 weights, diagonal Laplace and the other small-network Bayesian baselines cease to be applicable, leaving Adam as the practical comparison.The appendix cites Hessian cost and diagonal-Laplace limitations, and IVON instability, as reasons for excluding those baselines.Source: G LARGER-SCALE BNN EXPERIMENT · pp. 38–39
  • DMA’s message passing has linear cost in the number of weights per sweep and avoids Hessian computation, Monte Carlo sampling, and cavity division.The stated per-sweep cost is O(NW × N).Source: G LARGER-SCALE BNN EXPERIMENT · p. 39

G.1 SETUP

  • The larger-scale experiment increases the network from 83 to 1,932 weights and uses four hidden layers with four output channels.The architecture has widths (6, 12, 48, 24) and 1,500 training inputs.Source: §G.1 · p. 39

G.2 RESULTS

On the 1,932-weight, four-output network, DMA converges in three epochs and its predictive uncertainty widens outside the training range. The reported extrapolation calibration error is close to that of the smaller network.

G.2 RESULTS

  • DMA converges at epoch 3 with training NLL −0.73 nats/ex per output on the 1,932-weight network, with no numerical failures during training.The small baseline reaches −0.16 nats/ex; the Bayes-optimal value is −1.38 nats/ex at β=0.1.Source: §G.2 · p. 39, Table 6 · p. 39
  • DMA’s predictive uncertainty bands widen outside the training range, while Adam’s fixed ±2β band does not adapt.The figures show predictions across all four output channels.Source: §G.2 · p. 39, Figure 17 · p. 40, Figure 18 · p. 40
  • DMA trains in 0.72 s, less than half the 1.7 s for Adam and 2.4 s for AdamW, despite higher per-epoch cost.DMA uses 3 epochs, compared with 100 for each optimizer.Source: §G.2 · p. 40

G.3 CALIBRATION

  • On the larger network, DMA’s extrapolation calibration error is +0.026, close to the +0.01 reported for the 83-weight network.Diagonal Laplace is omitted at this scale because it does not scale to the network size.Source: §G.3 · p. 40, Figure 21 · p. 41
Loading 2609.29466v1…