PaperlayerView exact revision on arXiv

Source-linked AI summary

Large Language Bayes Is Not Reparameterisation-Invariant

Jian Xu

arXiv:2610.00265v1cs.LG

TL;DR

Large Language Bayes weights candidate probabilistic programs by exponentiated evidence bounds, but the bounds can assign different weights to programs encoding the same model. The paper measures this inconsistency, its consequences, and how often language models produce the relevant parameterisations, and examines a verification-based repair.

  • Problem

    LLB’s evidence-based model weights use an approximation that can vary with a program’s parameterisation, even though the marginal likelihood for equivalent programs is invariant.

    Source: §1 · pp. 1–2

  • Method

    The paper compares evidence bounds for equivalent parameterisations, studies their effects on model comparisons and language-model outputs, and tests template-generated reparameterisations verified by change of variables.

    Source: §1 · p. 2

  • Results

    Across families, dimensions, and funnel severities, equivalent-program weight discrepancies reach 31.9× and reverse sign; the published procedure still shows differences up to 8.9× at J = 64.

    Source: §1 · p. 2, §4.2 · p. 5

  • Takeaways & Limitations

    Statistical detection can merge genuinely different models, whereas verifying generated reparameterisations avoids that failure and closes the reversal window.

    Source: §1 · p. 2

    The repair equalizes scores only when the template library maps equivalent programs into a common verified candidate set.

    Source: §6.2 · p. 9

Abstract

from arXiv · show

Large Language Bayes (LLB) answers an informal modelling question by sampling candidate probabilistic programs from a language model, running approximate inference on each, and averaging them with weights proportional to an exponentiated evidence bound. We show that this weighting depends on how a model is written. The log marginal likelihood is invariant to reparameterisation; the evidence bound is not. On eight schools the centered and non-centered programs are the same measure to $5.7\times10^{-14}$, yet their weights differ by $6.1\times$; importance weighting reduces this only to $2.2\times$, and reproducing the inference LLB actually runs, a full-covariance Gaussian matched to the posterior moments, still leaves $1.9\times$ on eight schools and $8.9\times$ in $64$ dimensions. Across likelihood families, dimensions and funnel severities the discrepancy reaches $31.9\times$ and reverses sign, so no single writing is uniformly preferable. It inverts Bayes factors against eight natural competitors, and the induced error in the model posterior, and in any downstream target, is controlled by the spread $\Delta$ of the bound shortfalls through a known sharp Hilbert-distance bound. Across $360$ programs from six language models the parameterisation written ranges from $0\%$ to $100\%$ centered and is stable within a model. Detecting equivalent programs statistically can falsely merge genuinely different models at practical sample budgets; verifying reparameterisations we generate ourselves cannot, and closes the window.

1 INTRODUCTION

Large Language Bayes averages candidate-program posteriors using weights based on approximate evidence bounds, but those bounds can depend on how a model is written. The paper quantifies this inconsistency and examines its consequences and prevalence.

1 INTRODUCTION

  • LLB’s ideal program weight uses the marginal likelihood, but its implemented self-normalised sampler substitutes exp L(n), with L(n) a variational lower bound.The language-model proposal probability cancels in the algorithm actually run.Source: §1 · p. 1
  • Marginal likelihood is invariant to reparameterisation, whereas variational bounds can differ across coordinate systems describing the same measure.Centered and non-centered hierarchical models are a standard example of equivalent parameterisations with different approximate-inference behaviour (Papaspiliopoulos et al., 2007).Source: §1 · p. 1
  • On eight schools, two programs encoding the same measure to 5.7 × 10−14 differ by 6.1× in weight; the discrepancy survives importance weighting and LLB’s moment-matched proposal, at 1.9× on eight schools and 8.9× at J = 64.Source: §1 · p. 2
  • Across families, dimensions, and funnel severities, discrepancies reach 31.9× and reverse sign; Bayes factors invert against eight natural competitors.The spread Δ of bound shortfalls governs model-posterior error through a sharp Hilbert-distance bound.Source: §1 · p. 2
  • Across 360 programs from six language models, centered writing ranges from 0% to 100% and is stable within each model.The proposed repair verifies template-generated reparameterisations; statistical detection can falsely merge genuinely different models.Source: §1 · p. 2

2 RELATED WORK

Prior work studies language-model-assisted modelling, approximate inference under different parameterisations, and evidence bounds. This paper distinguishes its focus: coordinate-dependent bounds used as model weights can alter model selection, despite invariance of the target quantity.

2 RELATED WORK

  • LLM-proposed models have been weighted by marginal likelihood in Bayesian workflows, while evidence-estimation accuracy matters for model posteriors and downstream targets (Domke, 2025).This paper identifies broken invariance as a systematic, writing-dependent component of that error, rather than estimation noise.Source: §2 · p. 2
  • The paper’s distinction is that coordinate-dependent bounds used as model weights can change model selection, unlike prior work asking which parameterisation infers better for a fixed model.Centered and non-centered hierarchies’ differing approximate-inference behaviour is established (Papaspiliopoulos et al., 2007; Tan & Nott, 2013).Source: §2 · p. 2
  • Figure 1 contrasts identical models receiving different bounds with a repair that verifies template-generated reparameterisations and keeps the tightest verified bound.Source: Figure 1 · p. 3

3 BACKGROUND

LLB samples formal models from a language model and combines their target posteriors using evidence-based weights. Evidence bounds approximate the marginal likelihood, but their tightness and convergence depend on the proposal and importance weights.

3.1 LARGE LANGUAGE BAYES

  • LLB targets the posterior over models and targets given data, with model probabilities proportional to the proposal probability times the marginal likelihood.Sampling models from the language-model proposal and self-normalising cancels that proposal factor, leaving weights proportional to exp L(n), where L(n) bounds log p(x | m(n)).Source: §3.1 · p. 3
  • The concern is not whether proposal mass over-represents hypotheses with many language-model expressions; that is a separate question about the prior p(m | t).The importance sampler remains correct under that proposal prior.Source: §3.1 · p. 3

3.2 EVIDENCE BOUNDS

  • The importance-weighted evidence bound LK is non-decreasing in K and converges to log p(x) under integrability conditions, but infinite-variance importance weights can make convergence slow.At practical K, that slow convergence can leave the bound gap open.Source: §3.2 · p. 3

4 REPARAMETERISATION DEPENDENCE

Evidence bounds can assign different weights to programs that encode the same model, because variational bounds depend on parameterisation even though marginal likelihoods do not. Across the tested settings, the discrepancy persists under LLB’s full-covariance proposal, can reverse direction, and can invert model rankings.

4.1 CASE STUDY: EIGHT SCHOOLS

  • 5.68 × 10−14: across 2 × 105 points, the centered and non-centered eight-schools programs agree to double-precision rounding, yet their bounds can differ.The programs define the same measure, so any weight difference is an artefact of the approximation.Source: §4 · pp. 3–4, §4.1 · p. 4
  • Increasing K by 2000 reduces the importance-weighting gap by only 56%; heavy-tailed centered weights leave the same model weighted 2 to 6 times differently through K = 2000.The centered ESS is 1.7 × 10−5 to 1.2 × 10−4 of draws, versus 3.1 × 10−2 to 7.1 × 10−2 for non-centered; the authors do not claim no K could close the gap.Source: §4.1 · p. 5

4.2 THE PUBLISHED PROCEDURE, NOT JUST MEAN-FIELD

  • 1.3–1.9× on eight schools and 3.2–8.9× at J = 64: the centered and non-centered bounds still differ under LLB’s full-covariance, moment-matched proposal.The discrepancy grows with dimension; the J = 64 problem uses synthetic data.Source: §4.2 · p. 5, Table 2 · p. 5

4.3 BEYOND EIGHT SCHOOLS

  • 31.9×: from J = 8 to J = 64, the Gaussian hierarchy’s gap rises from 2.03 to 3.46 nats; the discrepancy’s sign also reverses across models and data settings.Centered bounds are tighter by 0.73 nats for hierarchical logistic models, 0.88 nats for Poisson models, and 1.37 nats for the Gaussian hierarchy when σ = 4.Source: §4.3 · p. 6

4.4 THE REVERSAL WINDOW

  • 0.805 nats at K = 2000: the reversal window contains eight of nine natural competitors, allowing worse-supported models to rank above the centered-scored hypothesis.At K = 1, the 2.100-nat window can invert Bayes factors up to 8.16×; at K = 2000, it can invert them up to 2.24×.Source: Figure 3 · p. 7, §4.4 · p. 7
  • The model-posterior total-variation error, and hence error in any downstream target, is at most tanh(∆/4), attained, where ∆ is the spread of bound shortfalls.At the measured gaps, the error is 19.9% to 48.2%; a common shortfall cancels, so repairing one model need not improve the ensemble unless it changes ∆.Source: §4.4 · p. 7

5 HOW OFTEN DOES THIS FIRE, AND WHICH WAY?

The authors measured how often language models write parameterisations that receive different evidence-bound weights, and found that the choice varies sharply across models while remaining consistent within each model. The programs were classified but not run through complete LLB pipelines, so these results measure the frequency of the writing, not how often a finished run is materially wrong.

5 HOW OFTEN DOES THIS FIRE, AND WHICH WAY?

  • The study prompted six language models for three hierarchical problems and classified 360 NumPyro programs using their parameterisation structure.Twenty independent completions were sampled per model per problem; programs without a hierarchy or matching neither classification pattern were excluded from the ratio.Source: §5 · pp. 7–8
  • The programs were classified but not run through a complete LLB pipeline, so the measured frequency does not establish how often a finished LLB run is materially wrong.The experiment also used NumPyro and a bare instruction, rather than LLB’s Stan setup with worked examples, K = 25, and 104 outer replicates.Source: §5 · p. 8
  • 0% to 100%: centered-program shares range from claude-opus-5 (0 of 57 classified programs) to gpt-4o-mini (47 of 47).Table 4 reports counts for 360 generated programs; “other” and “no hier.” programs are excluded from the share.Source: §5 · p. 8, Table 4 · p. 8
  • Each model writes nearly the same parameterisation across all three problems, with claude-opus-5 non-centered and gpt-4o-mini centered throughout.The authors describe parameterisation as a stable model habit rather than a response to the problem.Source: §5 · p. 8
  • The centered share falls across the tested OpenAI versions from 100% for gpt-4o-mini to 50.9% for gpt-5.5, but the latest model still writes the centered form about half the time.The authors note that mixed parameterisations make a hypothesis’s weight depend on which program is sampled, so repeated runs can give different model posteriors.Source: §5 · p. 8

6 REPAIR

The paper finds that statistical tests on prior-predictive samples can miss practically important model differences, so it instead generates and verifies candidate reparameterisations. This avoids false merges, but the guarantee depends on the template library and exact bounds; Monte Carlo estimates introduce a further qualification.

6.1 DETECTION IS UNRELIABLE AT PRACTICAL BUDGETS

  • The proposed detection test compares prior-predictive samples using forward sampling, avoiding inference, but no threshold separated the tested Normal and Student-t(30) models.The comparison was evaluated under a permutation null using kernel MMD rejection rates.Source: §6.1 · pp. 8–9
  • At tested dimensions and sample sizes, Normal versus Student-t(30) likelihoods had MMD rejection rates of 0.00–0.10, while true reparameterisations were never distinguished and grossly different models always were.The errors are asymmetric: missing an equivalence changes nothing, whereas merging different models destroys a real Bayes factor.Source: §6.1 · p. 9
  • The equivalence-detection difficulty is about realistic budgets, not impossibility: characteristic-kernel MMD is consistent against any fixed pair of distinct distributions.The authors note that two models may agree in aggregate while differing at the observed data, where their evidence matters.Source: §6.1 · p. 9

6.2 VERIFICATION, NOT DETECTION

  • The repair generates candidate reparameterisations with transformations and log-Jacobians, checks them numerically against equation 3, and scores each model using its tightest verified candidate.False merges are structurally impossible because each candidate is compared only against the program that generated it.Source: §6.2 · p. 9
  • −32.116 to −31.326: verification lifts the centered eight-schools bound, shrinks the reversal window from 0.805 nats to 0.015, and restores all eight reversed competitors.The probe check separated correct from corrupted transformations by fourteen orders of magnitude; the validity guarantee rests on the templates, not on the probe check establishing identity everywhere.Source: §6.2 · p. 9
  • Equal scores for reparameterised programs are conditional on the template library mapping both into a common verified candidate set, and noisy Monte Carlo maxima do not exactly inherit the lower-bound guarantee.At the reported sample sizes, estimation noise is two orders of magnitude below the gaps at issue; deployment should match estimation cost across candidates and take the maximum with a margin.Source: §6.2 · p. 9

7 CONCLUSION

The conclusion is that evidence-bound weights can depend on how a model is written, with consequences for Bayes-factor and model-posterior comparisons. The paper also reports rerunnable code and detailed experimental specifications.

7 CONCLUSION

  • The same model can receive weights differing by 2.2 to 6.1× on eight schools and up to 31.9× in higher dimensions, with the direction depending on the model and data.The paper reports that this can invert Bayes factors against eight natural competitors and that model-posterior error is governed by the spread of bound shortfalls across candidates.Source: §7 · p. 9
  • Across six tested language models, the written parameterisation ranged from never to always centered and was stable within each model; statistical detection was unreliable at practical budgets, while verifying generated reparameterisations closed the window.These conclusions concern the tested programs and verification procedure described in the paper.Source: §7 · p. 9

REPRODUCIBILITY STATEMENT

  • The authors state that the experiments are rerunnable and provide code, generated programs, raw results, and detailed specifications in the supplementary material.Fits were deterministic given reported seeds, and ground-truth marginal likelihood calculations used closed-form marginalisation and one-dimensional quadrature.Source: REPRODUCIBILITY STATEMENT · p. 10

A LIMITATIONS

The paper isolates parameterisation-dependent evidence weighting from disputes about the prior over models. Its empirical scope and proposed verification have specific boundaries: the study covers selected Gaussian settings, and the verifier relies on verified templates and finite probes.

A LIMITATIONS

  • The diagnosis concerns replacing invariant marginal likelihoods with coordinate-dependent variational bounds, not whether the language model’s prior over models is desirable.The paper also leaves malformed or non-hierarchical generated programs outside this diagnosis; it reports that roughly 6% are in those categories.Source: A LIMITATIONS · pp. 11–12
  • The verifier checks transformations at finitely many probe points, relying on symbolic verification of its templates rather than proving arbitrary transformations sound.An engineered deviation on a set of negligible probe-measure could pass; a symbolic checker on program text would be stronger.Source: A LIMITATIONS · p. 12
  • The empirical study uses mean-field Gaussian inference across three likelihood families, four dimensions, four funnel severities, and one three-level hierarchy.It does not cover mixtures, state-space models, Gaussian-process latents, or discrete latent structure; exact analyses also remain tied to Gaussian cases.Source: A LIMITATIONS · p. 12
  • The repair covers location-scale hierarchies, not marginalisation-versus-augmentation equivalences or other transformations without their own templates and symbolic arguments.The frequency study covers three problems and six tested models, but only one model from each of two providers; unmatched programs were excluded, so percentages indicate which parameterisation dominates rather than precise rates.Source: A LIMITATIONS · p. 12

B WHAT THE GAP COSTS

The spread of evidence-bound shortfalls controls both Bayes-factor reversals and the error in the normalized model posterior. In practice, parameterisation habits can change rankings across practitioners, while the direction of the discrepancy depends on the problem.

B WHAT THE GAP COSTS

  • A log Bayes-factor error equals the difference between the two models’ bound gaps, and LLB reverses a comparison when the true log-evidence margin lies within that gap difference.For a centered hierarchical model against a competitor with a tight bound, the reversal window is its bound shortfall: 0.805 nats at K = 2000 and 2.100 at K = 1.Source: B WHAT THE GAP COSTS · pp. 12–13
  • The model-posterior total-variation error is at most tanh(∆/4), a sharp bound using the spread ∆ of bound gaps.The paper identifies ∆ with the Hilbert projective distance between the true and computed weight vectors and specializes the known sharp Hilbert-distance result.Source: B WHAT THE GAP COSTS · p. 13
  • 19.9% to 48.2% is the reported worst-case model-posterior total-variation error, and the same bound applies to the final downstream answer.These values use measured gaps with competitor bounds taken as tight, at K = 2000 and K = 1, respectively.Source: B WHAT THE GAP COSTS · p. 13
  • Reducing individual gaps does not necessarily improve the global guarantee, because repairing only the largest gap can increase their spread.For example, gaps (10, 9) have spread 1, while changing them to (0, 9) increases the spread to 9.Source: B WHAT THE GAP COSTS · p. 13

WHAT THIS MEANS IN PRACTICE

  • Two practitioners using different APIs can obtain different rankings for the same model, while one practitioner’s runs can vary when a model splits its writing habits.The paper says which effect is worse depends on the decision receiving the posterior.Source: WHAT THIS MEANS IN PRACTICE · p. 13
  • The direction of the discrepancy depends on the problem: centered writing penalizes weakly identified Gaussian hierarchies but can tilt comparisons the other way elsewhere.The paper reports sign reversals for other likelihoods and for informative observations about the group-level scale.Source: WHAT THIS MEANS IN PRACTICE · p. 13

PROOFS

The proofs connect differences in evidence-bound shortfalls to comparison reversals and posterior error. They establish a sharp total-variation bound, extend it to downstream targets, and report numerical checks of the stated results.

PROOFS

  • A Bayes-factor error is exactly the difference between two bound gaps, and a ranking reversal occurs when the true log-evidence margin falls between zero and that difference.The proof compares normalized true and approximate weights, whose normalizing constants cancel in a ratio.Source: PROOFS · p. 14
  • The model-posterior error is bounded by tanh(∆/4), with equality attainable in the limit for a two-group configuration.The extremal configuration places mass p⋆ = 1/(e^(∆/2) + 1) on the group with the smallest gap.Source: PROOFS · p. 14
  • The total-variation error for downstream target mixtures is no greater than the error between their model weights.The proof uses that each model-specific target is a probability measure; numerical checks report no violations and match the extremal bound to four decimal places across seven ∆ values.Source: PROOFS · p. 14

C VERIFICATION, NOT DETECTION

The repair generates reparameterisations from templates and verifies them against their source program rather than testing arbitrary pairs for equivalence. It tightens bounds for covered models, but its guarantees depend on the templates and do not make LLB generally reparameterisation-invariant.

C VERIFICATION, NOT DETECTION

  • The algorithm generates reparameterisations from templates, checks the change-of-variables identity at probe points, fits variational inference to admitted candidates, and returns their tightest bound.Candidates are checked against the program that generated them, not against other hypotheses.Source: C VERIFICATION, NOT DETECTION · pp. 14–15
  • Because candidates are checked only against their source program, the procedure cannot merge two distinct models; a verification failure costs one candidate, unlike a detection failure that can cost a Bayes factor.The statistical detection failure mode is therefore structurally absent from this design.Source: C VERIFICATION, NOT DETECTION · pp. 14–16
  • At S = 2 × 10^5 probes, a violation on probe-measure 10^-4 escapes with probability under 2 × 10^-9, but small regions can evade checks and affect high-density posterior areas.The templates—not finite probe checks—provide validity; probes guard against misapplication and implementation errors.Source: C VERIFICATION, NOT DETECTION · p. 15
  • Verified reparameterisations share evidence, so the maximum of their lower bounds tightens a model’s bound without overshooting its evidence; pairwise rankings can still move either way.The guarantee assumes every candidate satisfies the change-of-variables identity for all latent values.Source: C VERIFICATION, NOT DETECTION · p. 15
  • The probe check separates the correct transformation’s 5.7 × 10^-14 violation from a smallest corrupted-case violation of 4.45, while relying on templates for validity.Tested corruptions included a missing Jacobian, an incorrect Jacobian exponent, a wrong scale map, and a 1% shift change.Source: C VERIFICATION, NOT DETECTION · pp. 15–16
  • 0.805 nats shrinks to 0.015, and all eight reversed competitors regain their correct ranking after the repair.The centered eight-schools bound rises from −32.116 to −31.326, within 0.015 nats of the quadrature evidence.Source: C VERIFICATION, NOT DETECTION · p. 16

SCOPE OF THE REPAIR

  • The repair targets centered location-scale hierarchies; nonlinear transformations, constraints, covariance alternatives, and marginalisation-versus-augmentation may remain unrepaired.A program outside the library keeps its current parameterisation-dependent bound, making the miss a silent no-op rather than a wrong answer.Source: SCOPE OF THE REPAIR · p. 16

D EXPERIMENTAL DETAILS

The experimental details describe the evidence calculations, inference procedures, equivalence tests, and language-model generations used in the study. They also document the models’ distinct roles as experimental subjects and as research aids.

D EXPERIMENTAL DETAILS

  • The eight-schools study uses mean-field Gaussian variational inference in unconstrained coordinates, while quadrature supplies a reference log evidence of −31.3113.The bounds use 4,000 outer replicates at each of six K values; the change-of-variables check uses 2 × 10^5 diffuse-Gaussian probes.Source: D EXPERIMENTAL DETAILS · pp. 16–17
  • The competitor models have analytic Gaussian evidences, isolating the comparison between tight competitor bounds and a looser hierarchical bound.Table 5 reports the centered hierarchical score of −32.116 at K = 2000 against true log evidence of −31.3113.Source: D EXPERIMENTAL DETAILS · p. 17, Table 5 · p. 19
  • The prior-predictive equivalence test uses quadratic MMD with an RBF kernel, 200 permutations, and 20 repetitions per table cell; Normal versus Student-t(30) is never separated.The tests use α = 0.05; rejecting equivalent pairs is harmless, while failing to reject distinct pairs is a false merge.Source: D EXPERIMENTAL DETAILS · p. 17, Table 6 · p. 19
  • The verifier is tested at 2 × 10^5 probe points with tolerance 10^-8 against deliberately corrupted transformations, including missing and incorrect Jacobians.The correct transformation’s maximum violation is 5.68 × 10^-14.Source: D EXPERIMENTAL DETAILS · p. 17
  • The breadth sweep covers Gaussian, hierarchical logistic, and hierarchical Poisson likelihoods, varying dimension, funnel severity, and hierarchy depth.Table 7 reports bound gaps and importance-weight tail-shape estimates across these configurations.Source: D EXPERIMENTAL DETAILS · p. 17, Table 7 · p. 20
  • Across 20 completions on each of three hierarchical problems, models’ parameterisation choices span never to always and remain stable within a model across problems.Recency softens, but does not eliminate, the pattern.Source: Figure 4 · p. 18, D EXPERIMENTAL DETAILS · p. 18
  • The six named language models generated all 360 programs analyzed, with completions classified as returned and without filtering, editing, or regeneration.The classifier applies semantic rules for non-centered and centered programs, excluding unmatched hierarchical programs from reported shares.Source: D EXPERIMENTAL DETAILS · p. 18, F USE OF LARGE LANGUAGE MODELS · p. 19

F USE OF LARGE LANGUAGE MODELS

  • Language models also aided literature review, manuscript wording, and proof drafting; authors checked cited primary sources and verified the proofs.The authors take responsibility for the final content, including text produced with generative-AI assistance.Source: F USE OF LARGE LANGUAGE MODELS · p. 19
Loading 2610.00265v1…