PaperlayerView exact revision on arXiv

Source-linked AI summary

Uncovering and Fixing Collider Bias in Bayesian PINNs

Michael Obermayr, Robert Peharz

arXiv:2610.11737v1cs.LGcs.AI

TL;DR

B-PINNs commonly couple physics and trajectory parameters through residual likelihoods, a structure that can bias physical-parameter posteriors even with priors centered on ground truth. The paper proposes a physics-to-trajectory chain model and discretizes its stochastic dynamics for particle MCMC; experiments report that the chain avoids the collider bias, subject to the method’s stated scope limits.

  • Problem

    B-PINNs use residual likelihoods to couple independent physics and trajectory parameters, raising the question of whether this dependency structure yields reliable physical-parameter posteriors.

    Source: §1 · p. 1

  • Method

    The paper models physics as generating trajectories and observations, then discretizes the stochastic dynamics so particle MCMC can sample the chain posterior.

    Source: §2 · p. 3, §2.1 · p. 3

  • Results

    Despite priors centered on ground truth, B-PINN posteriors systematically underestimate oscillator dissipation and stiffness, while the chain accurately recovers their identifiable ratios.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The paper identifies two null conditions under which collider B-PINNs yield unbiased posteriors: a parameter-independent residual metric and no dissipation bias.

    Source: §6 · p. 8

    The particle MCMC scheme targets stochastic ODEs in low-to-moderate dimensions; particle filters degenerate as σr approaches zero and face weight collapse on finely discretized PDEs.

    Source: §6 · p. 8

Abstract

from arXiv · show

Bayesian physics-informed neural networks (B-PINNs) are a popular framework for parameter and state inference from sparse or noisy observations. They are commonly formulated via a collider structure, in which physical and trajectory parameters are assumed to be a priori independent and become coupled through virtual likelihoods on differential-equation residuals that enforce physical consistency. We show that this modeling choice can induce severe systematic bias in the posterior over physical parameters: even when the prior is favorably centered on the ground-truth parameters, the resulting posterior can drift away and concentrate far from them. As a remedy, we advocate a hierarchical chain model in which physics generates trajectories, which in turn generate observations. The chain model does not suffer from this posterior bias, but it poses a harder, so-called doubly intractable, inference problem due to a physics-dependent normalization constant. This challenge can be resolved by discretizing the underlying stochastic dynamics, after which the chain posterior can be sampled exactly with particle MCMC. We identify two distinct mechanisms characterizing the collider bias, derive analytical approximations of their magnitudes, and establish diagnostic criteria for predicting when standard B-PINNs remain reliable. Experiments confirm the predicted bias and show that the chain formulation successfully avoids it.

1 INTRODUCTION

B-PINNs couple initially independent physics and trajectory parameters through equation-residual likelihoods, while the alternative chain model makes trajectories depend directly on physics. The paper reports that collider posteriors can drift from ground truth even under favorable priors, motivating a chain formulation despite its harder inference problem.

1 INTRODUCTION

  • In B-PINNs, independent priors over physics and trajectory parameters become coupled only when conditioning on residuals treated as zero-valued virtual observations.The residual likelihood is typically a narrow, zero-mean Gaussian.Source: §1 · p. 1

Which of the two formulations is preferable?

  • Despite priors centered at ground truth, the collider underestimates oscillator dissipation and stiffness, while the chain accurately recovers their identifiable ratios.The collider bias recurs across resampled datasets.Source: §1 · p. 2
  • The chain instead models physics as generating trajectories, which in turn generate observations, but requires a physics-dependent normalization constant.The chain posterior is doubly intractable because calculating that constant requires marginalization over trajectory parameters.Source: §1 · p. 2, Which of the two formulations is… · p. 2
  • The paper identifies the collider’s normalization constant as the source of posterior drift, favoring less restrictive physics and potentially yielding confidently wrong estimates.The authors note that the collider remains computationally simple despite this bias.Source: Which of the two formulations is… · p. 2

2 PROBLEM FORMULATION AND PROBABILISTIC MODELING

The paper models uncertain second-order dynamics with process noise, sparse noisy observations, and latent trajectories. It compares a collider model, which uses residuals to couple parameters, with a chain model that normalizes a physics-conditioned trajectory distribution.

2 PROBLEM FORMULATION AND PROBABILISTIC MODELING

  • Unknown physical parameters govern a second-order ODE through an invertible generalized mass matrix and force field, while the trajectory is typically latent and observed only sparsely and noisily.The stated system class includes applications in classical mechanics, structural dynamics, and robotics.Source: §2 · p. 3
  • Process noise turns the deterministic trajectory into a distribution; the model represents it with an SDE whose noise enters velocity increments through Aλ.Aλ may depend on state and physical parameters.Source: §2 · p. 3
  • Choosing Aλ specifies how noise enters: Aλ = Id models acceleration noise, whereas Aλ = Mλ models stochastic forces or torques and can contribute to collider bias.The residual scale Σr is fixed as a hyperparameter rather than inferred with λ.Source: §2 · p. 3

2.1 Physics-Informed Residuals and Energy

  • At uniformly spaced collocation points, Brownian increments imply residual covariance Σr/h, yielding a quadratic energy that measures trajectory compatibility with physical parameters.This energy is the shared ingredient in the probabilistic formulations.Source: §2.1 · p. 3

2.2 Probabilistic Models: Collider vs. Chain

  • The collider gives trajectory parameters an independent prior and treats residuals as pseudoobservations, so the parameters interact only through conditioning on zero residuals.The residual likelihood’s Gaussian normalization is invariant to λ and θ.Source: §2.2 · p. 4
  • The chain follows λ → θ → y and defines trajectories conditionally on physics by tilting a reference density toward low-residual trajectories.Normalizing this conditional distribution generally requires a constant that depends on λ.Source: §2.2 · p. 4
  • The model difference is the collider bias: the collider posterior is reweighted by Zλ, which favors physics admitting a larger volume of low-residual trajectories.This reweighting can displace the posterior from data-generating parameters and make credible intervals miss ground truth.Source: §2.2 · pp. 3–4

3 ANALYSIS OF THE COLLIDER BIAS

The analysis approximates collider bias for neural trajectory surrogates and separates it into residual-metric and dissipation mechanisms. Their effects depend on how noise enters the dynamics and on the system’s drift.

3 ANALYSIS OF THE COLLIDER BIAS

  • A Laplace expansion around the data-free mode decomposes the approximated collider bias into residual-metric and dissipation terms, explaining its origin and direction.The expansion uses Gaussian weight priors and the Gauss–Newton curvature of residual energy.Source: §3 · p. 4
  • The approximation relies on a near-ODE-fitting surrogate, weakly λ-dependent prior penalty, locally stable constrained rank, additive decoupling, piecewise-constant dynamics terms, and sufficient surrogate bandwidth.These are simplifying assumptions for the bias decomposition.Source: §3 · pp. 4–5
  • The residual-metric bias arises because the collider omits the Jacobian determinant of the acceleration-to-residual mapping, whereas Zλ accounts for it across neff constrained directions.Acceleration residuals avoid this mechanism; force residuals add a term involving −neff times the log determinant of Mλ.Source: §3 · p. 5
  • Along drift-invariant parameter directions, the residual-metric artifact can overwhelm finite-data scale information and push posterior mass toward vanishing masses.Its magnitude grows with neff.Source: §3 · p. 5
  • The neural surrogate’s nonlocal temporal coupling produces a dissipation bias that penalizes dissipation in dissipative systems, with error growing as Θ(T).For growing modes, the analogous penalty targets stronger instability.Source: §3 · p. 5

4 TRACTABLE INFERENCE IN THE CHAIN MODEL

The chain model’s physics-dependent normalization makes inference doubly intractable. Discretizing the stochastic dynamics yields a Markov state-space model with analytic per-step normalizers, enabling particle MCMC inference for the exact posterior.

4 TRACTABLE INFERENCE IN THE CHAIN MODEL

  • Discretizing the generative SDE factorizes the chain normalization into analytic per-step terms, enabling asymptotically exact inference with particle MCMC.This avoids nested variational inference or Langevin re-estimation of Zλ.Source: §4 · p. 5

4.1 The Chain Model as a State-Space Model

  • Semi-implicit Euler–Maruyama discretization gives a Markovian trajectory prior with independent increments and analytic per-step normalization.Together with the physical-parameter prior and observations, it defines the chain model over λ and the state path.Source: §4.1 · p. 5

4.2 Exact Collider Bias on the State-Space Latent

  • On the state-space latent, the drift acts as a mean shift and the dissipation bias vanishes; acceleration residuals make the posteriors coincide, while force residuals retain mass bias.The remaining bias under force residuals points toward vanishing mass determinants.Source: §4.2 · pp. 5–6

4.3 Exact Parameter Inference via Particle MCMC

  • An unbiased particle-filter estimate of p(y | λ) costs O(PT); substituting it into Metropolis–Hastings lets PMMH target the exact posterior for any P ≥ 1.Sampling a trajectory alongside accepted parameter values also yields exact joint samples.Source: §4.3 · p. 6

5 EXPERIMENTS

Across three dynamical systems, the chain model recovers the truth while the collider fails where the analysis predicts. The experiments isolate residual-metric and dissipation mechanisms, test when collider bias vanishes, and evaluate corrections and inference schemes.

5.1 Systems and Setup

  • The evaluation covers a damped oscillator, a double pendulum, and a relativistic particle, comparing state-space and neural-surrogate inference across acceleration and force residuals.The setup also includes a corrected Fourier-feature B-PINN; its closed-form correction is evaluated only on the oscillator.Source: §5.1 · p. 6

5.2 Collider Bias and Its Removal

  • With acceleration residuals on the Markovian state-space latent, the collider and chain are bit-identical because the normalization correction is constant.This is the null condition in which both bias mechanisms are inactive.Source: §5.2 · pp. 6–7
  • The chain model recovers the truth in every system, while the collider fails where the analysis predicts.Table 1 reports the chain’s drift-invariant scale within 1.1 posterior standard deviations and identified parameters within 1.3; collider failures match the predicted cases.Source: §5.2 · p. 6, Table 1 · p. 7
  • Force-residual metric bias rewards smaller masses, overwhelming process-noise constraints and driving mass estimates downward; the pendulum B-PINN also shrinks both rod lengths.On the oscillator state-space latent, the Kalman posterior collapses to log α = −42.4; the pendulum rod estimates are 0.14 and 0.27 of their true values.Source: §5.2 · p. 7
  • Omitting the divergence correction systematically underestimates oscillator damping and overestimates the relativistic particle’s speed of light.Across ten resamples, damping was underestimated by 0.42±0.03 in log(c/m), with truth-covered 90% intervals on 4 of 10 datasets; the particle estimate was c = 1.78 versus 1.00.Source: §5.2 · p. 7
  • As the observation horizon grows, B-PINN damping estimates move farther from truth without posterior contraction, while the chain posterior contracts.Reweighting the chain posterior predicts the dissipation bias’s direction, growth, and 60–75% of its magnitude.Source: §5.2 · pp. 7–8, Figure 3 · p. 7
  • Adding the normalizer correction to the otherwise identical B-PINN removes 97.5% of oscillator damping bias under acceleration residuals and two-thirds of mass bias under force residuals.PMMH is also reported as up to 13× faster than B-PINN; NSVI converges to a biased oscillator damping ratio and NPSGLD fails to converge in the reported benchmark.Source: §5.2 · p. 8, Comparison with nested inference · p. 8

6 CONCLUSIONS

The paper identifies collider-induced posterior bias in B-PINNs and presents a Markovian chain formulation with particle MCMC to avoid it. Its reliability criteria and limitations bound when the approach applies.

6 CONCLUSIONS

  • Conditioning on virtual residual observations makes physics and trajectory parameters colliders, neglecting a parameter-dependent normalizer and biasing physical-parameter posteriors.The bias can persist despite priors centered at ground truth and can become more pronounced and overconfident with longer records.Source: §6 · p. 8
  • A Markovian state-space chain formulation with particle MCMC avoids collider bias, while analytical corrections can remain insufficient.The paper reports that its scheme avoids the nested Langevin overhead and variational gap of prior approaches.Source: §6 · p. 8
  • Collider B-PINNs yield unbiased posteriors when the residual metric is parameter-independent and dissipation bias is negligible across posterior support.Otherwise, the paper recommends the chain formulation, or a closed-form correction when the metric and drift divergence are state-independent.Source: §6 · p. 8
  • Particle MCMC is limited to stochastic ODEs in low-to-moderate dimensions; filters degenerate as process noise approaches zero and can suffer weight collapse on finely discretized PDEs.Choosing the process-noise covariance to prevent residual misspecification remains open; jointly inferring it introduces a third collider-bias mechanism left for future work.Source: §6 · p. 8
  • Collider bias is hardest to detect without ground truth and is most pronounced in weakly identified regimes and long horizons with sparse data.The paper advises checking the null conditions and preferring a chain formulation when they are violated.Source: §6 · p. 8

A.1 Laplace Approximation of the Collider Bias for Global Surrogates

The Laplace approximation separates collider bias for global surrogates into a residual-metric contribution and a drift-operator contribution. These mechanisms yield useful approximations, but their assumptions limit whether the correction recovers the chain posterior exactly.

Residual metric

  • The residual metric contributes a bias of −n_eff per unit increase in log κ_λ, with n_eff saturating at rank G_λ when physics dominates the prior on constrained directions.For a state-space latent with a flat prior, n_eff = Td = Nd exactly, making the metric term the exact collider bias.Source: §A.1 · p. 12

State-space latent

  • For a state-space latent, the metric bias is exact and has no dissipation term because the Markovian latent normalizes the path density step by step.A global surrogate instead gives the midpoint (Onsager–Machlup) form.Source: §A.1 · pp. 12–13

Drift operator

  • The drift-operator mechanism yields a leading dissipation term based on the integral of the absolute divergence, but growing modes are penalized like damped modes.When divergence changes sign along the orbit, the absolute value enters; the divergence alone does not determine the bias.Source: §A.1 · p. 13

Approximation limits

  • The global-surrogate decomposition is approximate: prior-dominated normalization, degenerate tangent maps, and non-additive rectangular factors require caveats.The correction captures dominant λ-dependence but adding −∆ to a B-PINN log-posterior need not recover the chain posterior exactly.Source: §A.1 · p. 13

Oscillator example

  • For a damped linear oscillator, dissipation penalizes larger c/m in proportion to T, while finite-bandwidth effects can still reward smaller k/m.The leading-order stiffness contribution vanishes, but its O(β/Ω) remainder grows with T.Source: §A.1 · p. 13

A.2 Exact Collider Bias along Drift-Invariant Directions

Along drift-invariant directions, the collider’s metric bias can be characterized exactly. Whether it appears depends on the residual metric and, under force residuals, it reweights the chain posterior toward smaller scales.

Acceleration residuals

  • With acceleration residuals, scaling masses and forces leaves residuals and observations invariant, so both chain and collider preserve the prior distribution of log α.This holds for any latent representation.Source: §A.2 · p. 14

Force residuals

  • With force residuals, the chain posterior depends on α through the process-noise level, while the collider adds an explicit scale reward.The collider posterior reweights the chain posterior by α^-Td for a state-space latent or α^-n_eff for a global surrogate.Source: §A.2 · p. 14

Scale of the bias

  • The collider’s force-residual reward grows with time steps or constrained directions, whereas the noise-level constraint is set by the finite observations.For the oscillator, the exact Kalman likelihood places the state-space collider at log α = −42.4.Source: §A.2 · p. 14

A.3.1 Setup

The experiments compare inference variants across systems and residual metrics while holding key settings fixed within each system. The setup also specifies sampler budgets and flags runs that do not yield valid posterior samples.

Evaluation grid

  • Each system combines acceleration and force/torque residuals with the listed inference variants, and within a system runs share data, prior, discretization, sampler budget, and surrogate settings.Changing one grid factor at a time isolates its effect, such as the residual metric.Source: §A.3.1 · p. 14

Systems and data

  • Data use the same semi-implicit Euler–Maruyama scheme as inference, at finer time grids; inference step counts and particle numbers are selected by convergence studies.The data grid is 2× finer for the oscillator and about 50× finer for the pendulum and relativistic particle.Source: §A.3.1 · pp. 14–15

Inference variants and sampling

  • The setup compares state-space chain and collider models, a standard Collider B-PINN, and an oscillator-only Fourier-MLP chain correction.PMMH and NUTS use distinct stated sampling budgets; runs without valid samples include force-residual state-space collider runs and non-mixed B-PINN chains.Source: §A.3.1 · pp. 15–16

A.3.2 Validation of the Reference Posterior

The authors validate the chain posterior reference by checking its likelihood estimator, discretization, and sampling behavior. The reported checks support its use as the reference for measuring collider bias.

Likelihood estimator

  • Particle-filter log-likelihood deviations at three oscillator points (+0.14, −0.10, +0.27 nats) are within Monte Carlo standard deviations of 0.27, 0.38, and 0.38.The authors conclude the likelihood estimator shows no bias at the resolution relevant to their comparisons.Source: §A.3.2 · p. 16

Simulation step

  • With data regenerated at a 32× finer step, exact chain-posterior displacements are −0.01 ± 0.07 in log(c/m) and +0.04 ± 0.02 in log(k/m).Across ten datasets, 90% credible intervals contain the truth in 8 or 9 datasets for every identified parameter.Source: §A.3.2 · p. 16

Alternative sampler

  • On the double pendulum and relativistic particle, NUTS reproduces PMMH posteriors and the collider’s collapse under torque residuals, while costing more per effective sample on fine grids.The authors use this comparison to check that results are not an artifact of the particle filter.Source: §A.3.2 · p. 16

A.3.3 Full Posterior Results

This section presents full posterior results across datasets, residual scales, and model configurations, examining collider bias and recovery of identified quantities. It also traces the bias to missing-normalizer effects and compares particle MCMC with nested inference methods, including their mixing, uncertainty representation, approximation fit, and computational cost.

A.3.3 Full Posterior Results

  • The chain model keeps the drift-invariant scale near truth and recovers every identified quantity within 1.3 posterior standard deviations, while collider errors follow the predicted parameter dependencies.Collider errors occur under force residuals for parameters entering the residual metric and for parameters entering the dissipation term, including oscillator damping and relativistic-particle light speed.Source: §A.3.3 · p. 17

A.3.4 Replication over Datasets

  • Across ten datasets, the B-PINN underestimates log(c/m) by 0.42 ± 0.03 and log(k/m) by 0.12 ± 0.03; chain-model intervals cover the truth in all ten.B-PINN intervals cover the truth in only 4 and 7 datasets, despite not being narrower than the chain’s. Exact posteriors on the linear surrogate reproduce the bias, while restoring the normalizer removes it, implicating the missing normalizer rather than dataset variation or surrogate expressiveness.Source: Table 7 · p. 18, §A.3.4 · p. 18

A.3.5 Sensitivity to the Residual Scale

  • With matched residual scale, collider bias persists and grows from −0.07 to −0.74 log(c/m) units, while both chain models remain near truth.At small scales, the displacement is still about one posterior standard deviation and coverage remains below nominal. Scale misspecification can also bias the chain; opposite-sign collider bias may then appear to improve coverage through cancellation, not correct modeling. The paper recommends matching the scale or jointly inferring it.Source: Table 8 · p. 19, §A.3.5 · p. 19

A.3.6 The Dissipation Term on the Conservative Systems

  • On conservative systems, the surrogate dissipation term is negligible for the double-pendulum scale but strongly favors larger light speed for the relativistic particle.For the pendulum, the term averages 3.05 nats at T = 12 without affecting the drift-invariant scale. For the particle, it is −5.18 nats at truth, with a +9.5-nat-per-log-c derivative; the B-PINN reports c = 1.78 versus 0.89 for the chain.Source: §A.3.6 · pp. 19–20

A.3.7 Restoring the Normalizer on the Neural Surrogate

  • On the neural surrogate with acceleration residuals, the normalizer correction removes 97.5% of the log(c/m) displacement, yet corrected k/m remains 3.48 versus truth 4.00.Source: §A.3.7 · pp. 20–21

A.3.8 Details on the Scaling-Law Experiments

  • As the record grows, B-PINN log(c/m) displacement worsens from −0.377 to −0.669 while its uncertainty barely changes; chain estimates stay near truth and contract.The leading-order dissipation correction predicts the direction and growth of bias but only three-quarters to three-fifths of its magnitude. Separately, the residual-metric term matches −T log α to machine precision, while the chain’s log-posterior changes by only 6.0 nats across the swept range.Source: §A.3.8 · p. 22

A.4 Comparison with Nested Inference

  • Particle MCMC yields one effective sample per 0.5–6.7 seconds, whereas NPSGLD costs 21–2,385 seconds per effective sample; NSVI is biased on the oscillator and fails to converge on the chaotic pendulum.More NSVI computation did not remove its oscillator bias, and its pendulum estimates were highly variable and overconfident, with standard deviations only 0.03–0.10 times those of particle MCMC.Source: Table 10 · p. 23, §A.4 · p. 23
  • Across all 18 runs, NPSGLD never reaches R̂ ≤1.05, while our method converges in every configuration at R̂ ≤1.001 and is asymptotically exact.NPSGLD reports pendulum m1+m2 = 1.495 with a standard deviation fourteen times too small and oscillator force-residual m = 1.000 with a standard deviation thirty times too small; identified-ratio posteriors are up to 2.5× too wide.Source: §A.4 · p. 24
  • Nested inference performs better on the linear oscillator than the chaotic pendulum, but NPSGLD costs more than 40× as much per effective sample as our method.The oscillator path posterior is close to Gaussian, while the pendulum posterior over 1,235 steps and two degrees of freedom is strongly non-Gaussian; with force residuals, oscillator process noise σr = 0.30 determines the drift-invariant scale.Source: §A.4 · p. 24
Loading 2610.11737v1…