PaperlayerView exact revision on arXiv

Source-linked AI summary

Amortizing Sequential Experimental Design in Reasoning LLMs

Anita Kriz, Antonin Berthon, Nicolás Astorga, Mihaela van der Schaar

arXiv:2610.20832v1cs.LGcs.AI

TL;DR

LLMs can struggle to adapt information gathering to experimental feedback, despite their knowledge and reasoning abilities. This paper trains reasoning policies for sequential BED using information-gain rewards evaluated under a model independent of the policy. The resulting policies improve information acquisition across unseen categories and transfer to Medi-Q, while short rollout horizons and contrastive-set coverage remain limitations.

  • Problem

    LLMs have been reported to perform poorly in multi-turn settings and to rely more on prior knowledge than on adapting reasoning to experimental feedback.

    Source: §1 · p. 1

  • Method

    The paper amortizes sequential BED into an LLM by training action-selection reasoning with information-gain rewards under an experimental model specified independently of the policy’s beliefs.

    Source: §1 · pp. 1–2

  • Results

    Amortized BED improves information acquisition across seen and unseen Twenty Questions categories and transfers positively to Medi-Q, a real-world clinical dialogue benchmark.

    Source: §1 · p. 2, §6 · pp. 9–10

  • Takeaways & Limitations

    The results suggest adaptive information gathering can be learned as a transferable reasoning capability rather than reconstructed through explicit search at every interaction.

    Source: §6 · p. 10

    The study uses relatively short rollout horizons, and performance is sensitive to contrastive hypothesis-set coverage when that set is misspecified.

    Source: §6 · p. 10

Abstract

from arXiv · show

Despite their extensive pretraining knowledge and strong reasoning capabilities, large language models (LLMs) struggle to adaptively gather information in experimental design tasks. We study how to amortize sequential Bayesian experimental design (BED) into an LLM by explicitly training the reasoning that guides action selection using information-gain rewards. Our approach specifies the experimental generative model independently of the LLM's own beliefs, allowing us to optimize expected information gain under a model of our choosing. We investigate the design choices for amortization, including alternative generative-model constructions, reward estimation, hypothesis-space specification, multi-step objectives, and computational costs relative to training- and inference-time BED baselines. On Twenty Questions, the resulting policies improve information acquisition and task performance, outperform inference- and training-time BED baselines at substantially lower computational cost, and generalize to unseen categories. We further demonstrate positive transfer from Twenty Questions training to Medi-Q, a real-world clinical dialogue benchmark. These results suggest that information gathering can be learned as a generalizable reasoning skill in LLMs.

1 INTRODUCTION

LLMs can rely more on prior knowledge than on adapting to new evidence, limiting adaptive information gathering. The paper trains reasoning policies with information-gain rewards and reports improved acquisition across unseen categories and transfer to Medi-Q.

1 INTRODUCTION

  • LLMs may rely more on prior knowledge than on adapting reasoning to new evidence, and have performed poorly in multi-turn settings.Prior work reports insensitivity to experimental feedback.Source: §1 · p. 1
  • The paper trains the reasoning process that guides action selection, formalizing information gathering as sequential Bayesian experimental design optimized with verifiable information-gain rewards.BED selects actions using an experimental generative model to reduce uncertainty about an unknown quantity.Source: §1 · p. 1
  • Training-time amortization allows the experimental generative model to be specified independently of the LLM’s beliefs, and the paper investigates rewards accumulated across interactions.The authors test whether multi-interaction information gain encourages non-myopic strategies.Source: §1 · p. 2
  • Amortized BED improves information acquisition across seen and unseen Twenty Questions categories and transfers positively to the clinical dialogue benchmark Medi-Q.The authors also report that decoupling the experimental model from the LLM’s beliefs improves information acquisition, and expected information-gain rewards improve learning relative to realized rewards.Source: §1 · p. 2
  • The study examines generative-model construction, reward estimation, hypothesis-space specification, multi-step objectives, and computational cost relative to BED baselines.The contributions include formulating amortized sequential BED for reasoning LLMs and studying its model and reward design.Source: §1 · p. 2

2 BACKGROUND & PROBLEM SETUP

Bayesian experimental design selects informative experiments, but sequential EIG optimization can be costly and myopic. Amortization shifts selection into a policy trained using trajectory-level information objectives.

2.1 BAYESIAN EXPERIMENTAL DESIGN

  • BED models an unknown quantity θ, design x, and outcome y, then selects designs by expected information gain under a prior and likelihood.Sequential designs update beliefs after each observation and condition the next choice on the interaction history.Source: §2.1 · p. 2
  • Sequential EIG optimization is costly because each decision requires posterior estimation and nested expectations, and one-step selection can miss designs with delayed information value.The posterior depends on the observed history, making the repeated estimation and optimization difficult to compute in advance.Source: §2.1 · pp. 2–3

2.2 AMORTIZING SEQUENTIAL EXPERIMENTAL DESIGN

  • Deep Adaptive Design amortizes sequential selection into a policy mapping interaction history to the next design, shifting EIG optimization cost to training.The policy can select designs at inference without repeating EIG optimization.Source: §2.2 · p. 3
  • DAD uses the equality between total trajectory EIG and summed conditional EIGs to train from sequential information gain rather than repeatedly simulating the expensive BED procedure.Its sequential prior contrastive estimation objective is a lower bound whose tightness improves as the contrastive set grows; prior-sampled hypotheses need not be re-estimated at each step.Source: §2.2 · p. 3
  • Prior work decomposes trajectory-level sPCE into per-step rewards, with a learned critic estimating future returns to support non-myopic policy optimization.The likelihood uses the product of outcome likelihoods across steps under conditional independence given the hypothesis and current design.Source: §2.2 · p. 3

3 RELATED WORK: BED WITHIN LARGE LANGUAGE MODELS

Prior LLM-based BED methods select designs at inference time or optimize policies at training time, while AR-BED explicitly trains reasoning and separates the training model from the policy. Table 1 organizes these modeling and training choices.

3 RELATED WORK: BED WITHIN LARGE LANGUAGE MODELS

  • Inference-time LLM BED methods estimate EIG from LLM-induced generative models, while CA-BED uses conversation-tree planning and ASIG amortizes selection into a policy.ASIG retains a generative model derived from the policy LLM’s predictive distributions.Source: §3 · p. 3
  • Table 1 compares policy structure and information-value estimation across prior methods and AR-BED, with indented rows marking ablations and shading marking changes.Its notation includes interaction history h and explicit reasoning trace z; candidate-proposal methods marked † use inference-time EIG selection.Source: Table 1 · p. 4
  • AR-BED explicitly optimizes reasoning before design selection, uses a training generative model independent of the policy LLM, and applies expected information rewards over multiple steps.The authors frame these choices as supporting non-myopic information gathering.Source: §3 · p. 4

4 METHOD: AMORTIZING BED IN REASONING LLMS

AR-BED trains an LLM to reason through hypotheses and select designs, evaluating information gain with a generative model independent of the policy. Expected rewards and multi-step returns provide training signals, with rollout length and hypothesis coverage remaining relevant constraints.

4 METHOD: AMORTIZING BED IN REASONING LLMS

  • The policy generates a reasoning trace before selecting a design, and GRPO trains the joint reasoning-and-design output using information-gain rewards accumulated over K-step rollouts.The training pipeline generates a design through reasoning, evaluates it with the separate generative model, and optimizes the policy.Source: §4 · p. 4

4.1 HOW SHOULD THE GENERATIVE MODEL p(θ, y; x) BE CONSTRUCTED?

  • The experimental generative model is specified independently of the policy, allowing a fixed training distribution that can include hypotheses the policy would be unlikely to generate.A separate likelihood model can provide more reliable outcome estimates than the policy’s predictive distribution when the initial policy is a poor approximation.Source: §4.1 · pp. 4–5
  • A fixed uniform prior over a finite contrastive hypothesis set defines the distribution for information-gain computation, while set size trades coverage against computational cost.The set’s composition and prior weights determine which hypotheses the policy is trained to distinguish.Source: §4.1 · p. 5
  • A separate semantic likelihood evaluator supplies distributions over finite outcomes, with conditional independence given the hypothesis and current design enabling likelihood accumulation across interactions.The outcome distribution is retained because an answer may be stochastic even when the hypothesis is known; likelihood evaluations can be cached and reused.Source: §4.1 · p. 5

4.2 HOW SHOULD THE REWARD BE ASSIGNED?

  • Marginalizing over hypotheses and outcomes yields a Rao–Blackwellized reward equal to one-step EIG over the specified finite hypothesis space, removing variance from sampling a single target and outcome.This equality applies to the specified finite space, not necessarily the full hypothesis space of interest.Source: §4.2 · p. 5
  • Multi-step returns reward designs for information gained later, but training trajectories from a smaller hypothesis subset than the contrastive set can make realized and expected rewards differ beyond variance reduction.The rollout horizon is limited to K steps for tractability; K = 1 is myopic, while K > 1 with γ > 0 includes future information.Source: §4.2 · p. 6

5 EXPERIMENTS

The experiments compare reasoning-based amortized BED with direct-policy, inference-time, and training-time alternatives. They examine information acquisition, hypothesis-space coverage, reward estimation, multi-step rewards, computational cost, and transfer to clinical dialogue.

5.1 AMORTIZING BED INTO REASONING IMPROVES GENERAL INFORMATION ACQUISITION

  • AR-BED1 achieved the best final solve rate across all categories and the highest average EIG per turn on Animals, Plants, and Countries.Figure 1 summarizes information acquisition and transfer across evaluation repeats.Source: Figure 1 · p. 7, §5.1 · p. 7

5.2 AMORTIZING BED INTO REASONING OUTPERFORMS INFERENCE- AND TRAINING-TIME BED BASELINES

  • BED-LLM was about 61× slower per game than AR-BED1, while ASIG was about 2.9× slower per training step.The comparison attributes these costs to online BED computation and the LLM-generated generative model, respectively.Source: Table 2 · p. 7, §5.2 · p. 8
  • AR-BED1 outperformed BED-LLM and ASIG on most domains, exceeding reasoning-ASIG on Animals, Plants, and Countries and matching it on Celebrities.BED-LLM improved only modestly over reasoning BASE; ASIG improved substantially with inference-time reasoning but remained below AR-BED1 on three domains.Source: §5.2 · p. 7, Figure 3 · p. 8

5.3.1 MISSPECIFIED PRIORS AND SIZE OF THE CONTRASTIVE SET

  • Mammals-only training performed worse than animals-trained and base models on held-out non-mammals, and worse than animals-trained on held-out mammals.Using a 200-sample contrastive set improved OOD transfer compared with a 100-sample set.Source: Figure 2 · p. 8, §5.3.1 · p. 8

5.3.2 USING EXPECTED OR REALIZED IG IN THE REWARD SIGNAL

  • Expected-reward training improved both EIG and solve rate on Animals, Plants, and Countries, but not on Celebrities, compared with realized-information-gain rewards.Source: §5.3.2 · p. 8

5.4 GENERALIZATION TO OTHER TASKS

  • After Twenty Questions training, accuracy rose from 90.5% to 94.7% on held-out dermatology diagnoses and from 84.8% to 90.2% on other specialties.The policies asked five questions in MediQ, with a fixed GPT-5 Patient and a separate GPT-5 decider.Source: Table 3 · p. 9, §5.4 · p. 9

5.3.3 USING A NON-MYOPIC REWARD SIGNAL

  • Multi-step rewards supported non-myopic information gathering in disease identification and attribute prediction, while open-ended questioning showed smaller lookahead benefits.In matched-state evaluations, K = 2 and K = 3 sometimes outperformed K = 1 at two- and three-question horizons.Source: §5.3.3 · p. 9

6 CONCLUSION

The paper presents amortized sequential BED as a way to train adaptive information gathering in reasoning LLMs. It reports transfer across unseen hypothesis spaces and to clinical dialogue, while identifying short horizons and contrastive-set coverage as limitations.

6 CONCLUSION

  • The approach decouples the experimental generative model from the policy and accumulates expected information-gain rewards across interactions.Source: §6 · p. 9
  • The method improves information acquisition across unseen hypothesis spaces and transfers learned behavior to Medi-Q clinical dialogue.The authors conclude that adaptive information gathering may be learned as a transferable reasoning capability rather than reconstructed through explicit search at each interaction.Source: §6 · pp. 9–10
  • The study uses relatively short rollout horizons, and performance is sensitive to contrastive-set coverage when that set is misspecified.The authors identify longer horizons with reliable training signals and the effects of set size, composition, and prior weighting as future directions.Source: §6 · p. 10

AI USE STATEMENT

The paper states that authors retained responsibility for its research and final interpretation while using generative AI for specified production tasks. Its related-work discussion distinguishes BED approaches by their estimation, amortization, and reward strategies.

AI USE STATEMENT

  • Generative AI assisted with coding implementation, figure editing, LaTeX formatting, and manuscript polishing; the authors determined and reviewed the research and take responsibility for the final content.Source: AI USE STATEMENT · p. 10

A EXTENDED RELATED WORKS

B.1 CONTRASTIVE SET ABLATIONS

The ablations show that contrastive-set size and coverage affect acquisition and transfer, while expected-reward training generally outperforms realized-reward training. The multi-step evaluations examine whether longer reward horizons support context routing, target prediction, and information acquisition.

B.1 CONTRASTIVE SET ABLATIONS

  • On Plants, expanding the contrastive set from 100 to 200 animals raises EIG from 0.260 to 0.268 and solve rate from 28.0% to 40.7%.On Animals, EIG is nearly identical at 0.330 nats, while solve rate is 43.3% versus 41.3%; the larger set also improves country EIG and solve rate, but celebrity EIG is essentially unchanged.Source: §B.1 · pp. 14–15, Figure 4 · p. 15
  • Restricting training to 100 Mammals instead of 100 diverse Animals lowers animal EIG from 0.330 to 0.245 and solve rate from 41.3% to 16.7%.Both metrics are also lower on all three out-of-distribution domains; this comparison changes both the contrastive set and sampled training targets.Source: §B.1 · p. 15

B.2 EXPECTED VERSUS REALIZED INFORMATION REWARDS

  • Expected-reward training raises animal EIG per question from 0.290 to 0.330 nats and solve rate from 34.0% to 43.3%, with gains on Plants and Countries but not Celebrities.Celebrities instead favor realized-reward training: solve rate is 23.3% versus 17.3%.Source: §B.2 · p. 15, Figure 5 · p. 16

B.3 NON-MYOPIC INFORMATION ACQUISITION

  • Multi-step rewards are evaluated in a controlled context-routing environment, target-attribute prediction during questioning, and information acquisition in Twenty Questions.The Cube-NM evaluation isolates learning an initially uninformative action; the other evaluations test whether longer horizons benefit natural-language policies.Source: §B.3 · p. 16

B.3.1 CUBE-NM: ACQUIRING CONTEXT BEFORE INFORMATION

In Cube-NM, a two-step policy learns to acquire context before disease information, even though the context test has zero immediate disease-information value. Related evaluations examine why this routing behavior requires a non-myopic objective and whether multi-step policies improve target-attribute prediction during questioning.

B.3.1 CUBE-NM: ACQUIRING CONTEXT BEFORE INFORMATION

  • After five tests, the two-step Cube-NM policy reaches 75.0% Bayesian MAP solve rate versus 30.2% for the one-step policy, despite trailing after the first test.The two-step policy selects the initially uninformative context test with almost unit probability; it overtakes the one-step policy after three tests. Results are matched single-seed outcomes, not estimates of training-run variability.Source: §B.3.1 · p. 16, Figure 6 · p. 17

B.3.2 WHY CUBE-NM REQUIRES NON-MYOPIC INFORMATION ACQUISITION

  • Because the context is independent of disease, its test has zero one-step information gain, but it reveals which disease-test block is informative.A two-step objective can value the context test through the information gained by the subsequent experiment, provided each context has an informative disease test.Source: §B.3.2 · pp. 17–18

B.3.3 PREDICTING TARGET ATTRIBUTES FROM THE QUESTIONING HISTORY

  • During target-attribute prediction, K = 3 improves medium-question accuracy and K = 2 improves hard-question accuracy over much of the trajectory.At turn five, overall accuracy is 32.7% for K = 3 and 30.8% for K = 2, versus 29.7% for K = 1; at turn 20, the values are 43.4%, 43.1%, and 43.8%, respectively. Policies generate their own histories, so the comparison does not isolate prediction under identical evidence.Source: §B.3.3 · p. 18, Figure 7 · p. 19

B.3.4 MYOPIC VERSUS NON-MYOPIC QUESTION SELECTION IN TWENTY QUESTIONS

In the simulated animal game, immediate- and downstream-information rankings often differ early, and lookahead selection trades immediate support reduction for fewer remaining hypotheses later. The finite Monte Carlo comparisons indicate opportunities for lookahead, but do not establish policy optimality.

Myopic versus non-myopic question selection

  • Open-ended questions can differ in both immediate EIG and how readily later questions distinguish surviving hypotheses, unlike the idealized balanced-query setting.The passage frames this as a possible source of non-myopic behavior, not an intrinsic property of natural-language tasks.Source: §B.3.4 · p. 19
  • The simulation samples animal-game states, tests four candidate first questions per state, and estimates longer-horizon EIG from eight GPT-5 continuations per candidate.The support analysis rescored saved questions and answers against a 200-animal contrastive set without regenerating continuations, so these are not exact optimal-planning values for that expanded set.Source: §B.3.4 · p. 19
  • Agreement between immediate- and two-to-five-question EIG rankings rises from roughly 20–40% at turns 0–2 to 60–70% at turns 8–10.Agreement remains below one, even later in the game.Source: §B.3.4 · p. 19, Figure 8 · p. 20
  • One-step selection leaves fewer hypotheses after the first question, while two- and three-step selection leave about 25.8 versus 28.0 after two questions and 18.0 versus 20.3 after three.These are candidate-selection diagnostics, not comparisons of trained AR-BED policies.Source: §B.3.4 · p. 19, Figure 9 · p. 20
  • Finite Monte Carlo estimates of candidate rankings and comparisons do not establish policy optimality.Declining average EIG alone also does not establish adaptive submodularity or myopic optimality.Source: §B.3.4 · p. 20

B.3.5 MULTI-STEP TRAINING MODESTLY IMPROVES DOWNSTREAM INFORMATION ACQUISITION

Multi-step training yields modest downstream information-acquisition gains on held-out animal-game states. The K = 2 policy has the largest mean gain at the three-question evaluation horizon, while longer training horizons do not improve acquisition monotonically.

Multi-step training modestly improves downstream information acquisition

  • Policies were compared on the same 180 held-out animal-game states, evaluating every training horizon over one, two, and three questions with γ = 0.5 for multi-step policies.Differences from K = 1 were paired by evaluation repeat; uncertainty is SEM across three repeats, not independently trained models.Source: §B.3.5 · p. 20
  • K = 2 changes average EIG by −0.0020, +0.0062, and +0.0116 nats per question at horizons one to three; K = 3 changes it by +0.0057, +0.0070, and +0.0080.The benefit is small and concentrated at longer evaluation horizons; K = 2 has the largest mean downstream gain despite slightly lower immediate EIG.Source: §B.3.5 · p. 20, Figure 10 · p. 21
  • Increasing the training horizon does not monotonically improve acquisition, and open-ended natural-language questioning offers smaller gains than the clear non-myopic strategy observed with Cube-NM.The comparison distinguishes learning a non-myopic strategy in a controlled environment from gains in open-ended questioning.Source: §B.3.5 · p. 21

C ADDITIONAL EXPERIMENTAL DETAILS

The additional experimental details specify the default Twenty Questions evaluation protocol and how solve rate and information gain are calculated.

Additional experimental details

  • Evaluations use 50 target games per domain and seed, a 20-turn budget, stochastic decoding, and three retries for malformed tool calls.Solve rate includes invalid games in the denominator; full-game information gain is cumulative realized gain divided by questions asked, averaged across games.Source: C ADDITIONAL EXPERIMENTAL DETAILS · p. 21

C.1 TRAINING DETAILS

This section documents the Twenty Questions and medical-policy training and evaluation setup, including prompts, hypothesis sets, the medical cohort, interaction protocol, and scoring metrics. It frames LLM reasoning and natural-language design choices as a trainable policy guided by an independently specified finite generative model and multi-step information rewards.

C.1 TRAINING DETAILS

  • Twenty Questions training uses animal-target games generated with GPT-5 and 1,200 sampled training states, with disjoint training and held-out evaluation targets and a distinct 200-animal contrastive set for likelihood rewards.The policies are LoRA adapters trained from unsloth/gemma-4-E4B-it; the setup uses GRPO with 8 rollouts per sample, 250 optimization steps, and evaluation repeats that vary target order or decoding seeds rather than training seeds.Source: §C.1 · p. 21

C.2 PROMPTS

  • The Twenty Questions prompt requires one yes/no/maybe tool question per turn, while proposal prompts request reasoning plus one XML-formatted question and oracle prompts estimate yes/no/maybe likelihoods for each guess.The MediQ policy prompt likewise requires one specific question per turn and prohibits providing the final diagnosis; a separate model makes that diagnosis after the question budget is exhausted.Source: §C.2 · pp. 21–23

C.3 DOMAIN AND TARGET LISTS

  • The listed Twenty Questions contrastive hypothesis sets comprise four domains: animals (N=200), plants (N=200), celebrities (N=100), and countries (N=195).Source: §C.3 · pp. 24–25

C.4.1 MEDICAL TQ TRAINING

  • The medical policy was separately fine-tuned on 1,200 synthetic states across dermatology, neurology, and gastroenterology using AR-BED1 expected information gain, with 130 training steps.Games used 50 disease labels, uniform starting weights, and up to 10 turns; states were sampled from valid prefixes of 304 GPT-5-oracle rollouts. This was a separate policy from the animal-trained model.Source: §C.4.1 · p. 26

C.4.2 BENCHMARK AND EVALUATION SUBSETS

  • The diagnosis cohort contains 269 MediQ cases: 129 filtered direct-diagnosis iMedQA cases and all 140 iCRAFT-MD cases, with 95 held-out-label dermatology and 92 out-of-domain cases in the main table.The held-out-label designation refers to the training contrastive set, not pretraining exposure; 54 dermatology, 15 neurology, and 13 gastroenterology vocabulary-overlapping cases are excluded.Source: §C.4.2 · p. 26

C.4.3 INTERACTION AND SCORING PROTOCOL

  • Policies complete five question–answer turns with a fixed GPT-5 Patient, while a separate GPT-5 decider evaluates the acquired history; reported metrics are argmax accuracy, correct-answer probability, and multiclass Brier score.Only the question policy changes between conditions, and uncertainties are means ± sample SEM across cases, not variation across training runs or Patient/decider samples; evaluation prompts are shared by both policies.Source: §C.4.3 · pp. 26–27, §C.4.4 · p. 27

C.4.4 EVALUATION PROMPTS

  • Clinical evaluation prompts require one specific question per turn, while a separate model answers from released patient facts and a final model returns probabilities for four options.The questioner must continue asking while the status is continue; the answering model returns relevant facts verbatim or a cannot-answer response.Source: §C.4.4 · pp. 27–28

D ADDITIONAL DISCUSSION OF THE DESIGN POLICY

  • Unlike conventional policies mapping states to predefined designs, an LLM policy selects natural-language designs and can be rewarded for the reasoning traces preceding them.The authors hypothesize that rewarding these traces could support more general information-gathering strategies.Source: D ADDITIONAL DISCUSSION OF THE DESIGN… · p. 29

D.1 CONSTRUCTING THE EXPERIMENTAL GENERATIVE MODEL

  • The training environment specifies its experiment model independently of the policy, allowing probability mass on hypotheses the policy might rarely generate.The chosen factorization is p(θ)p(y | θ, x), using a finite contrastive prior and a separate LLM semantic likelihood evaluator for open-ended designs and enumerable outcomes.Source: §D.1 · pp. 29–30

D.2 NON-MYOPIC INFORMATION RETURNS

  • The method Rao-Blackwellizes one-step information rewards over enumerable outcomes and estimates future value with finite-horizon policy rollouts optimized using GRPO.This avoids learning a critic over the large, structured natural-language state-action space; the rollout horizon and discount control return depth and future reward weighting.Source: §D.2 · p. 30

D.3 FIXED CONTRASTIVE SETS AS FINITE BAYESIAN MODELS

  • A fixed contrastive hypothesis set defines a finite training model in which posterior and predictive distributions can be computed exactly, with repeated likelihood evaluations cached.The posterior and predictive are exact with respect to the finite world, assuming the specified likelihood model is correct.Source: §D.3 · pp. 30–31

D.4 REALIZED VERSUS EXPECTED INFORMATION REWARDS

  • Expected rewards equal one-step EIG under the finite contrastive model, while marginalizing over hypotheses and outcomes removes sampling variance.Realized rewards have the same expectation when targets are sampled from the contrastive prior, but are single-sample estimates.Source: §D.4 · p. 32

D.5 WHEN THE TRAINING-TARGET DISTRIBUTION DIFFERS FROM THE CONTRASTIVE PRIOR

When training targets differ from the contrastive prior, realized-reward training can include a predictive-distribution disagreement term beyond information gain. Decoupling the generative model from the policy keeps the information objective fixed as the policy changes.

D.5 WHEN THE TRAINING-TARGET DISTRIBUTION DIFFERS FROM THE CONTRASTIVE PRIOR

  • When training-target and contrastive predictive distributions differ, expected realized reward adds a KL disagreement term to EIG_q; expected reward still marginalizes under pC.The added term vanishes when the two distributions match.Source: §D.5 · pp. 32–33
  • The training-target distribution q determines which histories are visited, while contrastive prior pC determines what counts as an informative design at each history.Source: §D.5 · p. 33

D.6 DECOUPLING THE GENERATIVE MODEL FROM THE POLICY

  • Decoupling the experimental generative model removes the policy’s influence on its information criterion, so policy updates change visited states and proposed designs but not what counts as informative.Source: §D.6 · pp. 33–34
Loading 2610.20832v1…