PaperlayerView exact revision on arXiv

Source-linked AI summary

Sample What You Say: Aligning Language Models to Sample the Distributions They State

Kasra Arabi, Virginia Smith, Chhavi Yadav

arXiv:2609.34929v1cs.LGcs.CL

TL;DR

Instruction-tuned language models can state a target distribution correctly yet sample from incorrect outcome probabilities, while prompting and decoding only partly reduce the mismatch. The paper introduces a per-rollout MMD witness advantage for GRPO; training substantially improves sampling on unseen distributions while largely preserving general capabilities.

  • Problem

    Instruction-tuned models can identify a specified distribution yet sample from it inaccurately, and prompting or decoding only partly addresses the mismatch.

    Source: §1 · pp. 1–2

  • Method

    The witness advantage estimates the negative MMD witness from leave-one-out group frequencies and uses it as per-rollout reward in GRPO.

    Source: §1 · p. 2

  • Results

    Training substantially improves sampling across unseen parameters and families and opinion-distribution and urn-draw tasks, while largely preserving general capabilities.

    Source: §1 · p. 3, §5 · p. 9

  • Takeaways & Limitations

    Group-defined distribution objectives need a mechanism that attributes group error to individual rollouts; this approach is studied for finite outcome spaces.

    Source: §5 · p. 9

    The paper studies finite outcome spaces; extending distribution-matching post-training to free-form text is identified as a next step.

    Source: §5 · p. 9

Abstract

from arXiv · show

Language models are increasingly used to sample from a specified distribution, for instance, to simulate survey respondents or generate synthetic data. Instruction-tuned models can state such a distribution correctly and still fail to sample from it. Prompting and changes to decoding reduce this mismatch only partly, which motivates training with policy optimization. Group relative policy optimization (GRPO) is a natural fit for this problem because it already samples a group of rollouts per prompt, and the group's empirical distribution can be compared with the target. However, scoring the group as a whole gives every rollout the same reward. Group-relative centering then sets all advantages to zero, and the model receives no learning signal. To give each rollout its own signal, we introduce the witness advantage, a per-rollout advantage derived from maximum mean discrepancy (MMD). It trains a model to match a target distribution over a finite set of outcomes. The MMD between the model's distribution and the target has a witness function that measures how over- or under-produced each outcome is. Each rollout's advantage estimates the negative witness at its outcome, so a rollout is rewarded for an outcome the group under-produces and penalized for one it over-produces. The witness advantage is computed in closed form from the group's outcome counts, and we use it as the reward in GRPO. On unseen target distributions, training with the witness advantage substantially reduces the total variation distance to the target while largely preserving the model's general capabilities.

1 INTRODUCTION

Instruction-tuned models can identify specified distributions yet sample from incorrect outcome probabilities, and prompting or decoding only partly corrects the mismatch. The paper proposes policy optimization with per-rollout witness credit so GRPO can adjust those probabilities toward the target.

1 INTRODUCTION

  • The model correctly identifies distribution families and parameters for five of six common families, yet samples remain far from the stated targets.For a coin with P(Heads) = 0.005, Qwen2.5-1.5B-Instruct assigns Heads probability 0.678; changing decoding can only partly correct the error.Source: §1 · p. 1
  • A group-level distribution reward gives all GRPO rollouts the same score, which group-relative centering converts to zero advantages.Subgroup scoring did not lower error, while supervised cross-entropy degraded general capabilities in the experiments.Source: §1 · p. 2
  • The witness advantage uses MMD’s witness to assign each rollout positive credit for under-produced outcomes and negative credit for over-produced outcomes.Its leave-one-out frequency estimate yields an expected update equal to the negative gradient of squared MMD before centering; centering adds bias of order 1/G.Source: §1 · p. 2
  • One to three GPU-hours of training substantially improve sampling on training distributions, unseen parameters and families, opinion distributions, and urn draws while largely preserving general capabilities.The paper also reports that prompting and decoding have limited success because the model’s probabilities over outcomes are already wrong.Source: §1 · pp. 1, 3
  • Prior group-scored GRPO approaches differ from this distribution-matching objective: GAPO assumes a uniform target, while distribution-aware reward targets a single numeric answer.GAPO’s fraction-of-group reward is biased because it counts the scored rollout itself; Jang et al. (2026) list group-scored rewards as future work but do not define or evaluate one.Source: §1 · p. 3

2 PRELIMINARIES

The paper represents a prompt as specifying a target distribution over finite outcomes and evaluates the model’s distribution over parsed outcomes. GRPO samples groups, scores individual responses, and centers their rewards to form advantages.

2 PRELIMINARIES

  • A prompt specifies a target distribution q over finite outcomes, while a parser maps each response to its stated outcome and invalid responses to ⊥.The model outcome distribution πθ(x) is the probability that its response states outcome x; the goal is πθ = q.Source: §2 · p. 3
  • Total variation measures the distance between model distribution p and target q, ranging from 0 to 1, and is estimated from n sampled outcomes when p is unavailable.TV is the largest difference in probabilities assigned by p and q to the same set of outcomes.Source: §2 · p. 3
  • GRPO samples G rollouts, assigns each a scalar reward, and defines its advantage as that reward minus the group mean.The update increases probability for positive-advantage responses and decreases it for negative-advantage responses; the reward affects the update through the advantage.Source: §2 · p. 3
  • For distribution sampling, every positive-target-probability outcome is valid, so correctness depends on how often outcomes occur rather than an individually checkable answer.An outcome is over-produced when its model probability exceeds its target probability and under-produced when it falls below it.Source: §2 · p. 3

3 METHOD

The method turns group-level distribution mismatch into per-rollout credit using the MMD witness, then uses GRPO to update the model. Its expected update follows the negative gradient of squared MMD before centering, while simpler group-scalar and sign-only rewards have distinct shortcomings.

3 METHOD

  • The method needs per-rollout credit derived from the group’s distribution mismatch, because GRPO requires a reward for each rollout although sampling correctness is a group-level property.The paper derives the witness advantage from MMD, analyzes its expected update, and compares it with group-scalar and sign-only rewards.Source: §3 · p. 4

3.1 THE WITNESS ADVANTAGE

  • The witness advantage estimates the negative MMD witness at each rollout’s outcome using leave-one-out group frequencies, rewarding under-produced outcomes and penalizing over-produced ones.With a finite outcome set and exact-match kernel, the witness is πθ(x)−q(x); leaving out the scored rollout removes a self-counting term of order 1/G that otherwise encourages concentration.Source: §3.1 · p. 4

3.2 THE WITNESS ADVANTAGE TRAINS TOWARD THE TARGET DISTRIBUTION

  • Before group-mean subtraction, the expected witness-advantage update is the negative gradient of MMD2, which is zero only when πθ = q.Centering scales that gradient by (G−1)/G and adds a concentration term of weight 1/G; with G ≥ 3 and unrestricted probabilities, the minimizer shifts by at most 1/(G−2) in TV, or 0.016 at G = 64.Source: §3.2 · p. 5

3.3 WHERE SIMPLER REWARDS FAIL

  • A group-scalar reward gives every rollout the same score, so GRPO centering eliminates its signal; scoring subgroups instead optimizes expected empirical TV, not the model’s own TV.For some targets, this objective is lower at a model that omits outcomes with q(x) < 1/G than at the target itself.Source: §3.3 · pp. 5–6
  • The sign witness follows the target direction except on low-mass outcomes, where removing mismatch magnitude can leave πθ = q nonstationary.At G = 64, an outcome with target probability 0.001 retains positive expected advantage until the model assigns it about 0.011.Source: §3.3 · p. 6

4 EXPERIMENTS

The experiments evaluate distribution matching on synthetic targets and real-world-like tasks, using excess total variation (TV) to compare sampling with a target. Witness-advantage training improves sampling across seen and unseen settings, including tasks closer to real applications.

4.2 RESULTS ON SYNTHETIC DISTRIBUTIONS

  • On five unseen distribution families, the witness advantage has the lowest median excess TV among compared methods and keeps general capabilities.Across those families, training removes 36% of excess TV; three quarters of the improvement remains when valid outcomes are removed from evaluation prompts.Source: Table 1 · p. 7, §4.2 · pp. 7–8
  • On non-Zipf training targets, witness-advantage training removes 79% of excess TV, from 0.477 to 0.101, while MMLU and IFEval remain within their noise bands.Supervised cross-entropy fits these targets better but lowers MMLU by 10 points and IFEval by 23 points; witness advantage changes them by 0.1 and 1.6 points.Source: §4.2 · p. 7
  • On 16 unseen-parameter targets outside Zipf, training removes 67% of excess TV, from 0.483 to 0.161; cross-entropy reaches 0.094 but costs capability.The unseen-family evaluation includes 100 targets from five families; prompts for unseen families list valid outcomes because the model cannot know outcomes of a never-seen family.Source: Table 1 · p. 7, §4.1 · p. 7, §4.2 · p. 8
  • The sign witness matches witness advantage outside Zipf, but on Zipf targets its median TV ends at 0.60 versus 0.10 for witness advantage.Zipf targets contain low-mass outcomes with target probability below about 0.016, which the sign witness does not train toward their targets; group-scalar reward ends at 0.55 on non-Zipf targets, where the untrained model starts.Source: §4.2 · p. 8, Figure 2 · p. 8

4.3 RESULTS ON TASKS CLOSER TO THE REAL WORLD

  • Training removes 80% to 92% of excess TV on urn and GlobalOpinionQA test prompts, 61% to 72% on unseen NYTimes prompts, and 56% to 69% on implicit variants.It also has lower excess TV than the best prompting method on every Figure 3 set; MMLU falls by 2.1 points while IFEval stays within its noise band.Source: Figure 3 · p. 9, §4.3 · p. 9

4.1 SETUP

  • Across opinion distributions and urn draws, training is evaluated on held-out prompts, an unseen NYTimes task, and implicit prompts without stated probabilities.The synthetic setup uses six discrete distribution families, 16 training targets per family except hypergeometric, and evaluates with excess TV, subtracting the expected finite-sample TV of a perfect sampler.Source: §4.1 · pp. 6–7, §4.3 · p. 9

5 DISCUSSION

The discussion argues that GRPO can reduce the gap between stated and sampled distributions when credit is assigned to individual outcomes, rather than through a shared group score. It situates this approach among sampling, diversity, and distribution-matching work, and identifies finite outcome spaces as the current studied scope.

5 DISCUSSION

  • Witness advantage converts group-level mismatch into per-rollout credit, improving sampling on synthetic and real-world-like tasks and transferring to unseen targets while largely preserving capabilities.The discussion argues that group-relative centering eliminates a scalar reward shared by every rollout.Source: §5 · p. 9
  • The study addresses finite outcome spaces; applying witness advantage to free-form text remains a next step, using example responses and a kernel to represent similarity.That extension would cover settings where outcomes cannot be enumerated.Source: §5 · p. 9

A RELATED WORK

  • Prior sampling remedies include inference-time prompting and supervised fine-tuning, while this work uses policy optimization with group-scored rewards and finds that advantages must vary by outcome and mismatch size.The related-work discussion distinguishes diversity restoration from matching a specific stated PMF, which may call for either more or less randomness.Source: A RELATED WORK · p. 15
  • Unlike methods targeting task-defined densities or scalar-reward distributions, this approach matches a stated PMF; its closed-form witness credit supports proofs of unbiasedness and mean-centering behavior.Related group-based methods include GAPO for uniform targets and distribution-aware reward for numeric rollouts against a scalar label.Source: A RELATED WORK · pp. 15–16

B NOTATION AND TOOLS FOR THE PROOFS

The appendix establishes notation for modeling outcome-level probabilities and group frequencies, then gives lemmas that simplify expected GRPO updates by working with outcomes and score functions.

B.1 NOTATION

  • The proofs model a finite outcome set X, target distribution q, model outcome distribution πθ, and groups of G independent rollouts.The parser maps responses to outcomes; invalid outcomes are included, with q(⊥) = 0.Source: §B.1 · p. 16
  • The leave-one-out frequency ˆp−i(x) counts outcome x among the G−1 rollouts other than i, while the group frequency includes all G rollouts.The appendix also defines total variation, MMD under the exact-match kernel, and centered advantages.Source: §B.1 · p. 16

B.2 FROM RESPONSES TO OUTCOMES

  • When rollout weights depend on responses only through outcomes, the expected response-level update equals the outcome-level update, though their individual updates and variances can differ.This reduction lets the proofs use outcome scores s(x) instead of response scores.Source: §B.2 · pp. 16–17

B.3 TWO LEMMAS ON THE SCORE

  • The score identities convert expectations involving an outcome score into gradients of outcome probabilities; pair moments isolate contributions from matches involving the scored rollout.Terms independent of that rollout’s score vanish because the expected score is zero.Source: §B.3 · pp. 17–18

C PROOFS FOR THE WITNESS ADVANTAGE

The appendix proves that the witness advantage makes the expected uncentered GRPO update descend MMD², then characterizes how centering and self-counting alter that update. It also formalizes the limitations of assigning one group-level reward to every rollout.

C PROOFS FOR THE WITNESS ADVANTAGE

  • For every group size G ≥ 2, the witness advantage gives E[U] = −∇θMMD²(πθ, q), and MMD² is zero only when πθ = q.Subtracting the group mean adds a bias term to the expected update.Source: C PROOFS FOR THE WITNESS ADVANTAGE · pp. 18–19

C.1 COUNTING THE ROLLOUT ITSELF

  • Counting a rollout in its own outcome frequency adds a 1/G concentration term, which pushes probability mass toward fewer outcomes.After centering, that term’s weight becomes 2(G−1)/G², and the stated TV shift bound doubles to 2/(G−2).Source: §C.1 · pp. 20–21

C.2 WHERE THE CENTERED UPDATE MOVES THE MODEL

  • The centered update’s unique minimizing distribution π⋆ is within 1/(G−2) in TV of q, but the appendix does not prove idealized-flow convergence to π⋆.A separate simulation over 45 targets came within 3 × 10−3 of π⋆ in every coordinate, with remaining distance shrinking over more steps.Source: §C.2 · pp. 21–22

C.3 A WORKED EXAMPLE

  • In the six-rollout example, the full-group variant gives the three a rollouts zero advantage even though the other five under-produce a relative to its target probability.The example contrasts full-group counts with the leave-one-out witness advantage.Source: §C.3 · p. 22, Table 2 · p. 22

C.4 NUMERICAL CHECKS

  • Numerical checks match the witness update to the automatic gradient within 1.5 × 10−8 and verify Equation 4 by enumeration to 3.4 × 10−15.At G = 8, the full-group update’s largest coordinate-wise deviation was 0.016, versus 0.001 for leave-one-out.Source: §C.4 · pp. 22–23

D.1 SETTING AND STATEMENTS

  • Giving every rollout the same group-scalar reward yields zero centered advantages; using subgroup rewards instead optimizes the biased surrogate FG, not the model’s own TV.Even sampling exactly from q incurs positive expected empirical TV from finite-group sampling error.Source: §D.1 · pp. 23–24

D.2 PROOFS

The proofs bound variability and bias for the subgroup TV reward, then simulate when that bias affects training. Numerical checks support the derived bounds and the reported scaling pattern.

D.2 PROOFS

  • Changing one draw alters subgroup TV by at most 1/G, giving Var(Tℓ) ≤ 1/(4G); the resulting within-batch advantage spread is bounded using this variance.The proof uses the independence of subgroup TVs and Jensen’s inequality.Source: §D.2 · p. 24
  • The subgroup-reward update performs expected gradient descent on FG, the expected TV of G draws, rather than on the model’s own TV.The proof keeps only the correlation between a subgroup’s TV and its own summed score.Source: §D.2 · p. 25
  • For targets with 64 rare tail outcomes, each of probability below 1/G, FG(q) = 0.1056 while the head-only point mass has FG = 0.1.Thus this surrogate strictly prefers deleting the rare outcomes to sampling from the target in that example.Source: §D.2 · pp. 25–26

D.3 WHEN THE BIAS MATTERS

  • Across five settings where the update ratio stayed at least 0.75 at the two largest simulated distances, κ ranged from 0.334 to 0.342, and the log-log fit against √((d−1)/G) had slope 1.01.The crossing is reported as proportional to the bias scale.Source: §D.3 · p. 26
  • In two settings the bias already mattered at the path’s start, and for (G, d) = (8, 100) and (64, 100), the update ratio stayed below 1/2 throughout.Where the Monte Carlo estimate exceeded five standard errors, its direction had cosine similarity at least 0.98 with −∇θTV(πθ, q); extending the uniform-target scaling to general targets is an extrapolation.Source: §D.3 · p. 26

D.4 NUMERICAL CHECKS

  • The appendix reports checks of the advantage-spread bound, the expected update against exact enumeration, the rare-outcome example, and the simulation.The quoted κ values and slope were computed against √((d−1)/G).Source: §D.4 · p. 27

E PROOFS FOR THE SIGN WITNESS

The sign witness approximates a direction that reduces total variation, but its behavior on low-mass outcomes limits when it can match the target. Simulations and a related property-ratio method clarify the scope of these results.

E.1 PART (I): THE POPULATION SIGN IS THE WITNESS OF TOTAL VARIATION

  • The population sign is the negative TV witness, and its expected update follows −∇θ 2TV(πθ, q) when model and target probabilities differ at every outcome.This identifies the direction of the update for the population sign, before considering its finite-group estimate.Source: E PROOFS FOR THE SIGN WITNESS · p. 27

E.3 PART (III): LOW-MASS OUTCOMES

  • If two low-mass outcomes have different target probabilities, πθ = q is not stationary when the model can assign any probability to each outcome.At stationary points, supported low-mass outcomes instead share a probability, regardless of their target probabilities.Source: E PROOFS FOR THE SIGN WITNESS · p. 27, §E.3 · p. 28

E.2 PART (II): THE SIGN COMPUTED FROM THE GROUP RARELY DIFFERS

  • For an outcome whose model and target probabilities differ by δ, the group sign disagrees with the population sign with probability at most exp(−2(G −1)δ2).Conditioned on the rollout's outcome, the leave-one-out count is binomial; Hoeffding's inequality bounds disagreement.Source: E PROOFS FOR THE SIGN WITNESS · p. 27, §E.2 · p. 28

E.3 PART (III): LOW-MASS OUTCOMES

  • For low-mass outcomes, the expected sign advantage is 2(1 −πθ(x))G−1 −1 and does not depend on q(x).A low-mass outcome has 0 < q(x) < 1/(G −1); the advantage is positive only below a model-probability threshold, even if the outcome already exceeds its target.Source: §E.3 · p. 28
  • For G ≥3, centering preserves the conclusion that supported low-mass outcomes with different target probabilities cannot both be matched at a stationary point.The expected centered advantage is a constant plus a strictly decreasing function of πθ(x).Source: §E.3 · p. 29

E.4 SIMULATING THE SIGN WITNESS ON ZIPF TARGETS

  • On three Zipf targets, the sign flow ends at TV 0.13 to 0.47, while the witness-advantage flow reaches TV below 10−4.The sign flow raises the low-mass outcomes' total probability to 1.5 to 2.4 times its target level.Source: §E.4 · p. 30

E.5 CONNECTION TO PROPERTY-RATIO ALIGNMENT

  • Outside a tolerance band, the binary property-ratio reward of Huang et al. (2026) is a shifted and scaled full-group sign witness; inside the band, it gives every rollout zero reward.Their binary targets do not enter the low-mass regime described in part (iii).Source: §E.5 · p. 30

F EXPERIMENTAL SETUP

  • Training prompts state distribution families and parameters, sometimes with a PMF; the strict parser maps invalid responses to ⊥, and target selection screens for estimable sampling error.A separate template is used for unseen-family evaluation; candidate targets are screened by a perfect sampler's expected TV at 5,000 draws.Source: §F.1 · pp. 30–31, §F.2 · p. 31, §F.3 · p. 31

F.4 TRAINING

This section specifies the GRPO training choices and how sampling error and general capabilities are evaluated. It also describes the supervised cross-entropy baseline and notes that its capability cost reflects the training recipe as well as the objective.

F.4 TRAINING

  • GRPO centers advantages by the group mean but does not divide by reward standard deviation, preserving the witness advantage's magnitude information.The training settings are shared across GRPO runs except where noted.Source: §F.4 · p. 32, Table 3 · p. 32
  • The cross-entropy baseline trains on target-weighted canonical response strings, with a 36-minute cap stopping its full fine-tune after 110 steps.Unlike GRPO, it has no KL term toward the initial model and uses five times the GRPO learning rate; its recipe was not tuned to preserve capability.Source: §F.4 · p. 32
  • The full fine-tune loses 10 MMLU points and 23 IFEval points, a cost attributed to both the recipe and the objective.The baseline was not tuned to preserve capability.Source: §F.4 · pp. 32–33

F.5 EVALUATION AND STATISTICS

  • Sampling error is median excess TV, with reductions computed from medians across targets; capability uses MMLU accuracy and IFEval instruction-level strict accuracy.Intervals are generally 95% paired bootstrap intervals over targets, with capability intervals resampling benchmark items.Source: §F.5 · p. 33

G.1 RESULTS PER FAMILY

Witness-advantage training improves sampling on unseen families, and the evaluation-format retrain improves all five. Comparisons also show differences across reward choices and that diversity-only reward can worsen sampling error.

G.1 RESULTS PER FAMILY

  • 0.615 to 0.623: occupancy is about unchanged after original-format training, while evaluation-format retraining improves all five unseen families.The original-format checkpoint improves four unseen families; the retrain improves all five.Source: §G.1 · p. 33, Table 5 · p. 34
  • On the three synthetic evaluation sets and general capabilities, supervised cross-entropy fits training targets and unseen parameters more closely than witness advantage, but loses 10 MMLU and 23 IFEval points.Source: Table 4 · p. 33

G.2 COMPARING THE THREE REWARDS

  • Across three seeds, witness advantage beats group-scalar reward on unseen parameters; the sign witness is indistinguishable from witness advantage on 36 non-Zipf unseen targets.On unseen families, intervals for witness advantage and group-scalar reward overlap at every tested group size.Source: §G.2 · p. 34, Table 6 · p. 34
  • 0.663 versus 0.477: entropy-bonus training raises median excess TV on non-Zipf training targets relative to no training.On unseen parameters, the corresponding medians are 0.730 versus 0.483.Source: §G.2 · p. 34

G.3 TRAINING-FREE BASELINES

Training-free baselines make limited progress on unseen families compared with witness-advantage training. Their outcomes vary: verbalized sampling struggles, while temperature tuning and renormalization offer bounded improvements.

G.3 TRAINING-FREE BASELINES

  • At most 19%: no training-free baseline removes more unseen-family excess TV, compared with 36% for witness-advantage training.The table evaluates methods on the untrained Qwen2.5-1.5B-Instruct model and reports reduction relative to that baseline.Source: §G.3 · p. 35, Table 8 · p. 35
  • 0.518: even the best verbalized-sampling setting has median list-to-target TV of 0.71, and the lists are often invalid or wrong.The best case pools ten verbalizations with a support-naming prompt; its parse rate is not the only issue, since parsed lists remain inaccurate.Source: §G.3 · p. 35
  • 0.546 versus 0.542: seeded and constant-seed prompts perform similarly at temperature 1, so seed randomness contributes nothing in this test.The tested seed conditioning is applied at inference only.Source: §G.3 · pp. 35–36
  • 0.600: renormalization slightly increases excess TV versus plain sampling, while target-tuned temperatures reach 0.478 and a shared temperature reaches 0.514.Both temperature choices are tuned with knowledge of the targets.Source: §G.3 · p. 36
  • 0.63 to 0.72 versus 0.59: each tested larger untrained model has higher median excess TV on unseen families than the 1.5B model.The comparison includes Qwen2.5, Llama-3.2, and Llama-3.1 instruction-tuned models.Source: §G.3 · p. 36

G.4 WHERE THE REMAINING ERROR IS

The diagnostics show that witness training mainly reduces coverage error, while residual errors differ between binomial and Poisson targets. On unseen families, most improvement persists without listing valid outcomes, although the list contributes.

G.4 WHERE THE REMAINING ERROR IS

  • Training lowers coverage error on both target families, but remaining binomial error is mostly shape while most Poisson error remains coverage.Coverage falls from 0.65 to 0.18 on binomial targets and from 0.95 to 0.57 on Poisson targets; the figure stacks median coverage and shape terms over four targets.Source: Figure 4 · p. 36, §G.4 · p. 37
  • Before training, coverage accounts for 71% of median binomial TV and 98% of median Poisson TV; Poisson samples have only 0.10 of the target mean.Median TV is 0.87 on binomial targets and 0.98 on Poisson targets.Source: §G.4 · p. 37

G.5 COPYING THE LISTED OUTCOMES

  • On the original hypergeometric prompts without listed outcomes, training changed excess TV from 0.625 to 0.614 and valid-output rate from 0.64 to 0.40.The prompts list valid outcomes because a model cannot know the support of a family it has never seen.Source: §G.5 · p. 37
  • 77% of the median improvement on unseen-family targets remains when the valid-outcome list is removed, although the list contributes.Median improvements are 0.176 with the list and 0.135 without it; their median per-target difference is 0.057 ([0.010, 0.103]).Source: §G.5 · p. 37
  • Without the list, log-series improvement is 0.226 ([0.204, 0.247]), but changing the prompt format prevents isolating the list's effect.The listed outcomes contribute, but most improvement does not depend on them.Source: §G.5 · p. 37

G.6 VARIETY OF THE TRAINING TARGETS

Training on more varied target families improves typical unseen-family performance, while training on a single target yields gains concentrated in related families and parameters. Because compute is matched, variety and prompt repetition change together.

G.6 VARIETY OF THE TRAINING TARGETS

  • Widening training from five to fifteen families lowers median excess TV on a typical unseen target by 0.038 (interval [−0.045, −0.025]).The comparison uses matched compute and evaluation-format prompts; Table 9 reports median excess TV over 100 unseen-family targets relative to the untrained model's 0.589.Source: §G.6 · p. 37, Table 9 · p. 38
  • The single-target run reaches pooled excess TV 0.397 versus 0.394 for fifteen families, but worsens hypergeometric and maximum-of-dice targets.Its gains concentrate on log-series (0.606 to 0.119) and occupancy (0.615 to 0.456); the five-family run improves every unseen family.Source: §G.6 · p. 37
  • On unseen geometric parameters, the single-family model has excess TV 0.02 to 0.11 for p from 0.155 to 0.914, while the single-target model reaches 0.44 at p = 0.155.The single-target model is accurate only near its training value.Source: §G.6 · p. 38
  • Matched compute confounds target variety with repetition: the single-target run gets 2,400 group visits, versus 10 visits per target for fifteen families.The comparison holds compute at 600 steps with the same group size and decoding.Source: §G.6 · pp. 37–38

H OPINION DISTRIBUTIONS AND STRUCTURED OUTPUTS

On opinion and structured-output tasks, trained models outperform the strongest prompting or decoding baselines on the reported comparisons. Training also improves sampling of sibling pairs, including both uniform and non-uniform targets.

H OPINION DISTRIBUTIONS AND STRUCTURED OUTPUTS

  • Opinion prompts state answer options and target probabilities, while the implicit variant omits the probability line and evaluation uses 250 draws per prompt.Training and test sets include urn-draw and GlobalOpinionQA prompts; no GlobalOpinionQA test question appears in training.Source: H OPINION DISTRIBUTIONS AND STRUCTURED… · p. 38
  • On opinion tests and NYTimes, the trained model has lower excess TV than the best prompting method; on opinion tests, it also beats the best temperature.Across seeds, paired margins are 0.14 to 0.32 over the best prompting method and 0.19 to 0.23 over the best temperature, with every interval excluding zero.Source: H OPINION DISTRIBUTIONS AND STRUCTURED… · p. 38, Table 10 · p. 38
  • Structured-output prompts name a parent and its children, then ask for one sibling pair; uniform targets omit pair probabilities, while non-uniform targets list them.The uniform task tests finding valid pairs, and the non-uniform task tests following stated probabilities.Source: H OPINION DISTRIBUTIONS AND STRUCTURED… · pp. 38–39
  • On uniform sibling targets, training removes 35% and 39% of excess TV across two seeds, versus 5% for the best decoding baseline and 8% for group-scalar reward.For non-uniform targets, training removes 47% of excess TV versus 14% for the best decoding baseline; on uniform targets, valid-pair coverage rises from 0.71 to 0.93 in the first seed.Source: Table 10 · p. 38, H OPINION DISTRIBUTIONS AND STRUCTURED… · p. 39
Loading 2609.34929v1…