PaperlayerView exact revision on arXiv

Source-linked AI summary

When Metropolis and Hastings Meet Bradley and Terry: Exact MCMC From Preference Voting

Ariel Smogorghevski, Nir Rosenfeld, Yaniv Romano

arXiv:2609.00905v1cs.LGstat.COstat.ML

TL;DR

Generative models make sampling easy, but rigorously ensuring samples come from the correct full target conditional distribution remains unclear, while MH requires pointwise density ratios. Pref-MH uses binary preference comparisons to sample exactly from judge-induced conditionals, and provides the first such MH framework for general state spaces and arbitrary proposal kernels.

  • Problem

    For generative models, rigorously ensuring samples come from the correct full target conditional distribution remains unclear, while MH requires pointwise density ratios unavailable from pairwise feedback.

    Source: §1 · p. 2

  • Method

    Pref-MH turns the MH–Bradley-Terry odds connection into an exact sampler using binary preference votes and user-specified proposal kernels.

    Source: Preview of Pref-MH and key contributions · p. 3, §1 · p. 3, §2 · p. 5

  • Results

    Pref-MH is the first exact MH framework using only binary pairwise preference comparisons, supporting general state spaces and arbitrary proposal kernels.

    Source: §1 · p. 3

  • Takeaways & Limitations

    Comparative feedback can directly define sampling distributions, especially where desired quality is difficult to reduce to a pointwise score.

    Source: §5 · p. 13

    Pref-MH assumes an underlying latent score follows the Bradley-Terry model, which may not hold exactly for LLM, VLM, or human judges.

    Source: §5 · p. 13

Abstract

from arXiv · show

Sampling from distributions conditioned on desired semantic properties is an emerging challenge in modern generative modeling. Metropolis-Hastings (MH) provides a principled route to conditional sampling, but requires access to exact pointwise target-density evaluations, which are not available in generative settings. Meanwhile, pairwise comparisons by humans or model "judge" are highly accessible and have proved valuable across diverse applications. We introduce Pref-MH, a general exact MH sampler for judge-induced conditional distributions using only stochastic binary pairwise comparisons. Our key observation is that the MH unnormalized density ratio matches the preference odds of the Bradley-Terry (BT) choice model. The central challenge is that while MH requires precise ratio computation, BT judges provide only sampled binary feedback. To this end, we develop a valid accept/reject rule whose resulting Markov chain provably converges to the target distribution. We further show that, for a fixed proposal kernel and budget, Pref-MH is optimal in the Peskun-Tierney sense among this class of exact reversible acceptance rules. Experiments on text generation and molecular design with LLM judges, as well as image generation with VLM judges, demonstrate that Pref-MH provides a practical and flexible approach to conditional sampling when comparative feedback is relatively easy to obtain.

1 Introduction

Pref-MH enables exact conditional sampling from judge-induced distributions using only stochastic pairwise votes, addressing the inaccessibility of pointwise target-density ratios in generative models. It connects MH ratios to Bradley–Terry preference odds and uses a randomized acceptance rule to preserve the target distribution.

1 Introduction

  • MCMC supports repeated, diverse samples, but generative models cannot ensure that prompted semantic properties hold or expose the exact conditional-density ratios MH requires.The introduction motivates this gap with text properties such as helpfulness, politeness, and Shakespearean style, and competing molecular-design criteria.Source: §1 · pp. 1–2

Preview of Pref-MH and key contributions

  • Pref-MH is an exact MH sampler for judge-induced conditional distributions that uses binary preference comparisons, general state spaces, and arbitrary proposal kernels.Exactness means the chain’s stationary distribution matches the desired target; the method does not require pointwise target-density evaluations.Source: §1 · p. 3
  • Under a Gibbs target, MH acceptance ratios match Bradley–Terry preference odds, linking density ratios to judges’ implicit preference strengths.The paper proves an equivalence between the ratios.Source: Preview of Pref-MH and key contributions · p. 3
  • With a fixed comparison budget, plug-in estimates cannot implement the oracle acceptance rule; Pref-MH instead randomizes acceptance to preserve the target distribution exactly.The paper proves the impossibility for any fixed number of binary comparisons and proves exact convergence for its randomized rule.Source: Preview of Pref-MH and key contributions · p. 3
  • For a fixed proposal and comparison budget, Pref-MH is Peskun–Tierney optimal among exact competitors and avoids scores or estimated acceptance ratios.Users can choose the number of judge queries per transition; increasing the budget can support higher acceptance probabilities and more exploration.Source: Preview of Pref-MH and key contributions · p. 3
  • The method combines independent judges for distinct properties, such as helpfulness and politeness, without requiring a manually weighted aggregate score.Proposal generation and comparative evaluation remain separate, making the sampler modular across feedback sources.Source: Preview of Pref-MH and key contributions · pp. 3–4
  • Pref-MH’s experiments span synthetic, text, image, and molecular-design tasks; the molecular-design experiment reports better external MolSkill than MH baselines without pairwise LLM judgments.MolSkill reflects a human-expert quality score and is not used to guide sampling.Source: Figure 1 · p. 2, Preview of Pref-MH and key contributions · p. 4
  • Unlike related methods tied to finite catalogs, specific protocols, or evaluable scores, Pref-MH supports general state spaces, user-chosen proposals, and fixed finite comparison budgets.The related-work discussion contrasts it with perfect sampling [34], human-in-the-loop MCMC [37], generative MCMC [38] [39] [40] [42] [43], and inference-time methods relying on base-model likelihoods [44].Source: Preview of Pref-MH and key contributions · p. 4

2 Problem Setup and Background

The setup defines a judge-induced target by reweighting a base distribution toward samples more likely to satisfy a desired property, while assuming only pairwise-comparison access. Pref-MH uses MH with a user-chosen proposal, but must estimate the unknown judge-likelihood ratio.

2 Problem Setup and Background

  • The target distribution reweights a base distribution p0 toward samples a judge considers more likely to satisfy property M.M can represent a single desideratum or multiple simultaneous desiderata, which the paper treats later.Source: §2 · pp. 4–5
  • The judge probability pJ(M | x) is unavailable directly; the assumed access is sampled pairwise comparisons between candidates.This matches settings where head-to-head judgments are available but each candidate’s true property probability is unknown.Source: §2 · pp. 4–5
  • MH proposes x′ from q(· | x), accepts or rejects it, and yields a chain with target π as its stationary distribution.The proposal kernel is user-specified and may range from resampling from p0 to local modifications, subject to usual convergence conditions.Source: §2 · p. 5
  • Although π cannot be evaluated directly, MH needs only target-density ratios; p0 and q are tractable, leaving pJ(M | x′)/pJ(M | x) as the challenge.The unknown judge-likelihood ratio is the remaining term to compute.Source: §2 · p. 5

3 Proposed Method: Exact Sampling from Preference-Based Judges

Pref-MH uses Bradley–Terry pairwise preferences to replace unavailable pointwise target-density ratios, then constructs finite-budget acceptance rules that preserve the desired distribution. The method extends to multiple judges and is optimal among exact rules under a fixed proposal and query budget.

3.1 Integrating Bradley–Terry within Metropolis–Hastings

  • Assuming pJ(M | x) ∝ exp(s(x)) and Bradley–Terry preferences, the MH target ratio becomes the baseline MH factor times the judge’s win-rate odds.This replaces pointwise evaluation of whether x satisfies M with pairwise comparisons between x and x′.Source: §3.1 · pp. 5–6

3.2 Impossibility of directly implementing the BT-MH acceptance rule

  • No finite fixed-budget procedure using binary votes can reproduce the oracle BT-MH acceptance probability for every unknown comparison probability p ∈ (0, 1).The result excludes degenerate moves whose acceptance is independent of judge outcomes; it motivates a different exact finite-budget rule rather than ruling out exact fixed-budget sampling altogether.Source: §3.2 · p. 6, The proof is given in Appx. D.1 · p. 6

3.3 A budget-optimal acceptance rule via N-vote construction

  • For every finite N, the N-vote rule yields a reversible Markov chain with target distribution π, and it is Peskun–Tierney optimal among exact rules using the same proposal, judge, and N queries.The vote count adjusts the baseline acceptance factor: unanimous votes for the proposal give the largest upweighting, while a roughly split vote leaves the factor close to 1.Source: §3.3 · p. 7, §3.4 · p. 9

The proof is given in Appx. D.2

  • Under standard Metropolis–Hastings proposal regularity assumptions, the Pref-MH chain converges to π from every initialization in its support for any fixed N ≥ 1.The assumptions ensure the chain can explore the target support and does not become trapped in a deterministic cycle.Source: The proof is given in Appx. D.2 · p. 8

3.4 Optimality

  • Among exact acceptance rules satisfying detailed balance and using the same proposal kernel and exactly N judge queries, the N-vote rule dominates competitors in Peskun–Tierney ordering for every proposed move.The comparison is conditional on the same proposal, judge, and query budget.Source: §3.4 · pp. 8–9

3.5 Composing multiple judges

  • For multiple desired conditions, Pref-MH multiplies the judges’ N-vote factors, adding each condition through its own judge rather than a hand-engineered score.The composite construction assumes conditional independence of the conditions given x and a Bradley–Terry model for each judge; with one judge it reduces to the single-judge rule.Source: §3.5 · p. 9

4 Experiments

Experiments validate Pref-MH in a controlled setting and evaluate multi-condition text generation, conditional image generation, and de novo molecular design; an additional machine-translation experiment appears in Appendix C.5.

4.1 Synthetic validation

  • Pref-MH continued approaching the known target distribution with one judge query per proposal, while plug-in MH plateaued at large error even with N = 4 queries.The synthetic comparison included an oracle MH with access to the latent score and an invalid plug-in sampler that substitutes empirical vote odds for the required ratio.Source: Figure 2 · p. 10, §4.1 · p. 10

4.2 Multi-condition text generation

  • Pref-MH with separate judges performed best across all three text properties under both evaluation metrics, while joint judging and pointwise-MH performed worse.The comparison used pointwise scores and BT coefficients; separate judges assessed each property, whereas joint variants assessed all three together, and the pointwise scores need not match the pairwise model’s latent signal.Source: Figure 3 · p. 11, §4.2 · p. 11

4.3 Image generation in continuous space

  • 63.6% (Pref-MH), 7.4% (Pointwise-MH), and 4.5% (base) of generated images contained all three target objects; aesthetic scores were 6.395, 6.289, and 6.287, respectively.The chain operates over continuous Gaussian noise vectors decoded by SDXL-Turbo and uses separate VLM judges for the orange salamander, turquoise butterfly, and red mushroom conditions.Source: §4.3 · p. 11, Figure 4 · p. 12

4.4 De novo molecular design from preference feedback

  • Pref-MH achieved the best MolSkill mean and median while maintaining high diversity and SA, without a major cost to other quality measures.Final states were evaluated after 1000 MCMC steps; Pref-MH used eight pairwise Qwen votes, with compute-matched pointwise scoring and a MARS [11] baseline using QED and SA.Source: §4.4 · pp. 12–13, Table 1 · p. 13

5 Discussion · A Additional Related Work · B Multi-judge Pref-MH with Exact N-vote Acceptance Algorithm

Pref-MH uses pairwise comparisons for exact conditional sampling, while its Bradley–Terry assumption may not hold exactly for real judges. The discussion frames comparative feedback as a way to define sampling distributions, and the related-work and appendix passages distinguish this approach and outline multi-judge voting.

5 Discussion

  • Comparative feedback can directly define sampling distributions when desired quality is hard to reduce to a pointwise score, though human judging may require efficient proposals to limit annotation burden.The authors also identify human experts as a possible source of comparisons and suggest studying how violations of the Bradley–Terry assumption affect the stationary distribution’s coherence.Source: §5 · p. 13

A Additional Related Work

  • Best-of-N typically returns one selected output and can rely on proxy rewards, while controlled-decoding methods rely on pointwise guidance; Pref-MH targets judge-defined conditional sampling from comparisons with convergence guarantees [58] [59] [14] [15] [16].The related-work passage also notes that multiple desired properties in best-of-N require choices such as scalarizing or combining objectives [60] [61] [62].Source: A Additional Related Work · p. 25
  • Unlike pseudo-marginal, exchange, and Bernoulli-factory MCMC methods, Pref-MH constructs accept/reject decisions from stochastic pairwise comparisons and preserves the target exactly for any fixed comparison budget.The passage distinguishes these approaches by their use of unbiased estimators, auxiliary draws, or random coins.Source: A Additional Related Work · p. 25
  • Human pairwise preferences inform alignment, and LLMs judge outputs, but Pref-MH uses comparative judgments for exact sampling and can guide one model with another or support multiple preference dimensions [21–25, 72, 26, 73–75].The passage also connects Pref-MH conceptually to self-rewarding, self-evaluation, and self-refinement while distinguishing its conditional-sampling procedure.Source: A Additional Related Work · p. 25

B Multi-judge Pref-MH with Exact N-vote Acceptance Algorithm

  • Algorithm 2 initializes a state and proposal kernel, queries each of multiple judges independently N times per proposed pair, and returns the resulting trajectory.The supplied pseudocode passages do not show the acceptance calculation.Source: B Multi-judge Pref-MH with Exact… · p. 26

C Experimental Details and Additional Results

The appendix gives the complete experimental setups and additional quantitative and qualitative results.

C Experimental Details and Additional Results

C.1 Synthetic validation

The synthetic experiment tests whether preference-only sampling recovers a known target without direct access to its score or density. Pref-MH approaches that target, whereas the empirical-odds plug-in sampler does not.

C.1 Synthetic validation

  • The controlled test uses a known, non-uniform base distribution and a multimodal target, requiring samplers to account for preference tilt and base-density ratios.The finite state space is X = {0, 1, . . . , 240}.Source: §C.1 · p. 26
  • Neither Pref-MH nor Plug-in receives the latent score or target density for accept/reject decisions; Pref-MH instead uses N binary judge outcomes, evaluated at N ∈ {1, 2, 4}.Oracle MH has direct access to the latent score and serves as a reference chain targeting π.Source: §C.1 · p. 27
  • Plug-in replaces Pref-MH’s vote-based construction with empirical win-rate odds and is not guaranteed to preserve the desired target.Chains start at k = 0 and run for 150,000 MH steps; the convergence plot retains burn-in samples to show behavior from the common nonstationary initialization.Source: §C.1 · pp. 27–28
  • Oracle MH and Pref-MH approach the true target, while equal-budget Plug-in plateaus away from it under the correctly specified Bradley–Terry comparison model.The methods use the same proposal and comparison budget; Figure 5 compares their empirical distributions with the target using N = 2 votes per proposal.Source: Figure 5 · p. 28, §C.1 · p. 28

C.2 Multi-condition text generation

This experiment tests whether preference-based MH can steer text generation toward three stylistic properties and compares separate property-specific judges with a single judge assessing their conjunction. Pref-MH with separate judges performs best across the reported criteria, with a particularly pronounced advantage on dialogue.

C.2 Multi-condition text generation

  • The experiment generates story openings from 50 scenario prompts, using a neutral system prompt, a 192-token limit, and temperature 0.9.The neutral wording is intended to prevent the base generator from being instructed directly to satisfy the hidden criteria.Source: §C.2 · p. 29
  • The target combines elevated literary language, gothic or dramatic atmosphere, and frequent direct dialogue, represented by either three property-specific judges or one joint judge.Separate-judge Pref-MH queries each judge three times per proposal; joint Pref-MH queries its single judge three times.Source: §C.2 · p. 29
  • The proposal mixes 0.10-probability independent draws from the base model with local suffix regeneration after a uniformly sampled cut point; the base-model factors cancel, leaving acceptance dependent only on judge terms.A cut point beyond the current response produces an unchanged proposal.Source: §C.2 · p. 29
  • Pointwise-MH baselines mirror the separate and joint judge decompositions but use normalized absolute ratings instead of binary pairwise comparisons.Separate pointwise MH uses three judges scoring from 1 to 10; joint pointwise MH uses one absolute-score judge for the full conjunction.Source: §C.2 · pp. 29–30
  • Pref-MH (sep.) performs best on every reported pointwise and pairwise criterion; its dialogue advantage over Pref-MH (joint) is especially pronounced.Both preference-based variants also outperform their pointwise counterparts, supporting the conclusion that the improvement is not explained by MH alone but by the comparative evaluation signal.Source: Table 2 · p. 30, §C.2 · p. 30

C.3 Image generation in continuous space · C.4 De novo molecular design

Pref-MH extends preference-guided sampling to continuous image latents and molecular generation, where it improves simultaneous image-attribute adherence and achieves the strongest reported MolSkill results. The experiments use specialized pairwise judges, with evaluations also assessing image quality and complementary molecular-design metrics.

C.3 Image generation in continuous space

  • In the image experiment, the chain evolves over SDXL-Turbo latent noise with a standard Gaussian prior, while Qwen3-VL-8B-Instruct judges each decoded image against a separate semantic criterion.The continuous-latent setup also tests combining specialized judges when prompting alone often yields only some requested attributes.Source: §C.3 · pp. 30–31
  • Image-generation proposals mix local pCN moves, wider pCN moves, and independent Gaussian refreshes with weights 0.6, 0.3, and 0.1, respectively.The pCN components use β values 0.08 and 0.25; each proposal component preserves reversibility with respect to the Gaussian prior.Source: §C.3 · p. 31
  • Image comparisons randomized candidate order and used nine votes per judge; Pointwise-MH instead used normalized absolute VLM scores, and the base drew independent Gaussian latents.Each MH chain ran for 3,000 steps, with samples saved every 300 steps for trajectory visualization.Source: §C.3 · pp. 31–32
  • 63.6% of images met all three target-object constraints with Pref-MH, versus 7.4% with Pointwise-MH and 4.5% with the base model; aesthetic scores were comparable, with Pref-MH slightly higher.The automated evaluation used Gemma-3-12B-IT and Aesthetic Predictor V2.5, neither of which guided sampling; Figure 6 shows additional trajectory samples.Source: Figure 6 · p. 32, §C.3 · p. 33

C.4 De novo molecular design

  • Molecular chains use valid SMILES states and local fragment edits; the LLM assesses medicinal-chemistry promise, including drug-likeness, synthetic accessibility, developability, and problematic motifs.The experiment follows MARS, uses its fragment vocabulary, initializes chains at CC, and limits molecules to 40 heavy atoms.Source: §C.4 · p. 33
  • For molecular proposals, MARS uses a QED–synthetic-accessibility objective, Pref-MH uses eight balanced-order binary votes, and Pointwise-MH averages eight integer ratings.All three methods use r0(x, x′) = 1 for valid proposals to isolate differences in guidance signals within the MARS editing framework.Source: §C.4 · pp. 33–34
  • Molecular evaluation uses 200 chains for 1,000 steps across five seeds and reports final-state QED, synthetic accessibility, diversity, and MolSkill; MolSkill is evaluation-only.MolSkill is a learned proxy derived from professional medicinal chemists’ pairwise preferences.Source: §C.4 · p. 34
  • Pref-MH achieved the best MolSkill mean and median while maintaining high synthetic accessibility and molecular diversity.The authors suggest pairwise LLM feedback captures medicinal-chemistry follow-up preferences not fully represented by MARS’s QED–SA objective or pointwise LLM scores.Source: §C.4 · p. 34

C.5 Machine translation

On WMT23 machine translation, Pref-MH leads on BLEU, chrF, and TER, while QUEST can score higher on XCOMET at smaller β. Pref-MH also outperforms a pointwise-judgment ablation across all four metrics in both directions.

Experimental setup

  • On WMT23, evaluation covers 549 German→English and 557 English→German source sentences, using TowerInstruct-7B-v0.2 as the base model at temperature 0.8.Chains start from ancestral samples, run for 128 steps, discard 32 burn-in states, and average scores over 16 retained states per source sentence.Source: §C.5 · pp. 34, 36

Sampling methods

  • All three samplers use suffix regeneration: retain a randomly selected prefix and regenerate the remaining translation from the base model, differing only in how quality enters acceptance.The proposal samples an index uniformly from the current translation's token positions; the proposal and base factors reduce accordingly.Source: §C.5 · pp. 34–35
  • Pref-MH uses five randomized-order pairwise judgments per proposal, while Pointwise-MH averages five 1–10 ratings with the same judge-query budget and cached current-state score.Both LLM-guided methods receive the same reference-free CometKiwi-XL quality estimate, and neither the human reference nor evaluation-only XCOMET-XL is available during sampling.Source: §C.5 · pp. 35–36

Evaluation and results

  • Pref-MH achieves the best BLEU, chrF, and TER in both directions, significantly beating every tested QUEST temperature and Pointwise-MH on all three metrics.Table 3 reports the chain-mean results after 32 burn-in steps and 16 retained states per source sentence.Source: Table 3 · p. 36, §C.5 · p. 36
  • QUEST scores higher than Pref-MH on XCOMET at smaller β, but loses that advantage as β rises; no tested QUEST setting improves XCOMET alongside any of BLEU, chrF, or TER.The authors interpret the trade-off as evidence that preference feedback is not reducible to maximizing the scalar QE reward, with gains across independent translation metrics.Source: §C.5 · p. 36
  • Pref-MH significantly outperforms Pointwise-MH on BLEU, chrF, TER, and XCOMET in both directions, despite matching its generator, proposal, QE information, judge, and query budget.This ablation isolates the role of comparative feedback.Source: §C.5 · p. 36

C.6 Compute resources and assets … D.2 Proof of Theorem 2

The appendix documents the experiments’ compute resources and assets, proves that no fixed-budget procedure can achieve the required acceptance probability for all judge preferences, and establishes single-judge reversibility and stationarity by specializing the multi-judge result.

C.6 Compute resources and assets

  • The experiments used an internal cluster with eight H200 GPUs or eight RTX PRO 6000 Blackwell GPUs per machine; synthetic validation ran on a two-core Intel Xeon CPU.The cluster machines had 224 or 112 Intel Xeon CPU cores and 2.0 TB RAM; the H200s had 143 GB memory each and the RTX PRO GPUs 96 GB each.Source: §C.6 · pp. 36–37
  • Reported experiment runtimes ranged from about 6 hours for image generation to 48 hours for molecular design, and no pretrained generative or judge model was fine-tuned.Machine translation took about 15 hours per direction, while multi-condition text took 10–12 hours on a single GPU; reported compute was dominated by inference, judge queries, sampling, and evaluation.Source: §C.6 · p. 37
  • The experiments used Llama-3.1-8B-Instruct, WMT23 data, SDXL-Turbo, Qwen3-VL-8B-Instruct, Qwen3-235B-A22B-Instruct-2507, and COMET/XCOMET under their specified licenses.The passage lists the Llama 3.1 Community License, WMT23 task terms, Apache-2.0 for both Qwen models, and corresponding Stability AI and Unbabel model-release licenses.Source: §C.6 · p. 37
  • Molecular design built on the official MARS implementation and fragment vocabulary [11] and evaluated molecules with the official MolSkill implementation of Choung et al..The MARS repository is under CC BY-NC 4.0, and MolSkill is under the MIT License.Source: §C.6 · p. 37

D.1 Proof of Theorem 1

  • A fixed-budget procedure using N binary judge outcomes and internal randomness cannot attain the required acceptance probability for every p when c > 0.Its marginal acceptance probability is a polynomial of degree at most N, whereas the required probability leads to a contradiction by polynomial identity.Source: §D.1 · pp. 38–39

D.2 Proof of Theorem 2

  • The single-judge N-vote chain is reversible with respect to π and has π as a stationary distribution, by specializing the multi-judge exactness result to m = 1.The proof uses the multi-judge construction with one judge.Source: §D.2 · p. 39

D.3 Convergence of the Pref-MH chain

Under proposal regularity, the marginal Pref-MH chain converges to π from every initialization in its support. The proof combines detailed balance with inherited irreducibility and aperiodicity, then extends convergence from almost every starting state to all states in the support.

D.3 Convergence of the Pref-MH chain

  • Assumption 1 requires a π-irreducible proposal and finite, strictly positive baseline ratios for proposed moves, with π-null sets receiving zero proposal probability from every state in the support.In the discrete case, the ratio condition requires positive finite ratios for every proposal with positive probability; when the base density is positive on the support, it is the usual forward–reverse support condition.Source: §D.3 · p. 40
  • Under the BT model’s finite-valued latent score, the judge requires no additional assumption for this convergence result.Source: §D.3 · p. 40
  • Pref-MH inherits π-irreducibility because each admissible proposal has positive marginal acceptance probability, so accepted proposal paths preserve the proposal’s reachability.Thus, proposal irreducibility implies π-irreducibility of the marginal Pref-MH kernel.Source: §D.3 · p. 41
  • Pref-MH is aperiodic because a proposed move has positive probability of rejection, giving the chain a positive probability of remaining at its current state.Together with π-irreducibility, this self-transition probability implies aperiodicity.Source: §D.3 · p. 41
  • The law of X_t converges to π from every initialization in the support of π.Detailed balance, π-irreducibility, and aperiodicity yield convergence for π-almost every initialization; the chain then reaches that full-measure set almost surely and remains in it.Source: §D.3 · p. 41

D.4 Proof of Theorem 3 … D.5.3 Multi-judge optimality

This section proves exactness and convergence for multi-judge Pref-MH and establishes that its N-vote rule Peskun–Tierney dominates every exact fixed-budget competitor, with single-judge optimality as a special case.

D.4 Proof of Theorem 3

  • Taking m = 1 in the multi-judge optimality result shows that the single-judge N-vote rule Peskun–Tierney dominates every exact fixed-budget competitor using the same proposal kernel and N queries.Its probability of moving from any state into any set of different states is at least as large as a competitor’s.Source: §D.4 · p. 42

D.5.1 Multi-judge exactness

  • Under the multi-judge Bradley–Terry assumptions, for every N ≥1 and valid proposal kernel q, the multi-judge N-vote rule satisfies detailed balance with respect to πm, making πm stationary.The resulting chain is reversible with respect to πm.Source: §D.5.1 · pp. 42, 45

D.5.2 Convergence of the multi-judge Pref-MH chain

  • Under Assumption 1, the multi-judge Pref-MH chain converges in total variation to πm from every initialization in supp(πm).Positive acceptance of admissible proposals gives irreducibility, while positive probability of a zero vote count gives a self-transition and aperiodicity.Source: §D.5.2 · pp. 45–46

D.5.3 Multi-judge optimality

  • For fixed proposal kernel q and N queries per judge, the multi-judge N-vote rule Peskun–Tierney dominates every exact detailed-balance competitor, with at least as high pointwise acceptance probability.The comparison applies to competitors using the same proposal kernel and exactly N queries from each judge per proposal.Source: §D.5.3 · p. 46
  • From every current state, the multi-judge N-vote sampler assigns at least as much probability as any exact fixed-budget competitor to moving into any set of different states.This satisfies the Peskun–Tierney domination condition.Source: §D.5.3 · p. 50

E Conditional Target Derivation

The section derives the multi-condition target by conditioning a base distribution on judge-defined events. Under conditional independence, it is proportional to the base density times the product of judge likelihoods, or, for exponential score likelihoods, the base density times the exponential of the summed scores.

E Conditional Target Derivation

  • The derivation assumes that the judge-defined conditions are conditionally independent given x, with each judge contributing its own likelihood factor pJi(Mi | x).The joint model draws x from the base distribution p0 and applies the associated judges to conditions M1, ..., Mm.Source: E Conditional Target Derivation · p. 50
  • The multi-condition target is p(x | M1, ..., Mm) ∝ p0(x) ∏_i pJi(Mi | x), and with pJi(Mi | x) ∝ exp(si(x)) it is proportional to p0(x) exp(Σ_i si(x)).This is the target used in the main text.Source: E Conditional Target Derivation · pp. 50–51

Related papers

Loading 2609.00905v1…