Amortized Thompson Sampling and Efficient Information-Directed Sampling for Simulator-Based Design

1 INTRODUCTION

• Thompson sampling selects an action optimal under a posterior-sampled hypothesis, but its repeated optimization burden is especially pronounced with continuous actions and costly, stochastic, or non-differentiable simulators.The paper situates this strategy alongside Bayesian optimization and information-directed sampling, which explicitly trades off expected regret and information about the optimal action (Russo & Van Roy, 2014a).Source: §1 · pp. 1–2, §1.1 · p. 3
• A parameter-conditioned policy combined with simulation-based inference produces approximate Thompson samples without repeating inference and reward optimization for each sample and round.The policy is trained offline to map unknown model parameters to approximately reward-maximizing actions, and the approach supports stochastic, non-differentiable simulators.Source: §1 · pp. 1–3
• A conditional action flow directly approximates the Thompson-sampling distribution from experimental histories, bypassing explicit parameter inference and providing tractable densities for information-gain estimation.Round-specific flows are trained offline over a prescribed horizon and conditioned at deployment on the evolving history.Source: §1 · pp. 2–3
• Expected information gain, immediate regret, look-ahead regret, and information-directed criteria can rank Thompson-sampled candidates or guide direct optimization over admissible actions.Direct optimization can find informative actions outside the support of Thompson sampling, while candidate selection favors plausible reward maximizers; information-seeking actions may incur immediate regret.Source: §1 · pp. 2–3
• The framework is demonstrated on synthetic-biology design problems involving fluorescent-protein production and a stochastic genetic-oscillator simulator.The applications involve uncertain experimental conditions and simulator-based optimization.Source: §1 · p. 3

1.1 BACKGROUND AND RELATED WORK

• Thompson sampling explores through posterior randomization, whereas information-directed sampling explicitly trades off regret and information about the optimal action (Thompson et al., 1933; Russo et al., 2018; Russo & Van Roy, 2014a).The paper also distinguishes its simulator-based approach from GP-surrogate information-search methods (Hennig & Schuler, 2012; Hern´andez-Lobato et al., 2014; Wang & Jegelka, 2017).Source: §1.1 · p. 3
• In continuous action spaces, top-two Thompson sampling's rejection rule does not identify meaningfully competing alternatives, motivating information-based selection or direct optimization to identify the optimal action.The paper separately learns an amortized posterior over latent parameters and maps posterior samples to optimal actions using a trained policy.Source: §1.1 · p. 4
• Unlike Bayesian optimal experimental design focused on unknown parameters, this work values information about the downstream optimal action.This focus can avoid acquiring parameter information irrelevant to identifying high-performing actions.Source: §1.1 · p. 4

2.1 PROBLEM FORMULATION

• The formulation treats simulator parameters as fixed across a trajectory and defines expected reward by averaging realized reward over observations.Conventional implementations repeat posterior inference and reward optimization as observations accumulate.Source: §2.1 · p. 5

2.2 AMORTIZED THOMPSON SAMPLING THROUGH PARAMETER INFERENCE

• A parameter-conditioned policy is trained to maximize prior-averaged expected reward using score-function gradients and PPO, without requiring simulator or reward derivatives.At deployment, its conditional mean approximates the optimal action for a parameter value.Source: §2.2 · p. 5
• A round-specific parameter flow approximates the posterior, and its samples are passed through the pretrained policy to generate approximate Thompson samples.This replaces per-history inference and per-sample reward optimization with pretrained-model evaluations.Source: §2.2 · p. 5

2.3 DIRECT AMORTISATION WITH AN ACTION FLOW

• An action flow directly learns the conditional distribution of approximately optimal actions from simulated parameter-history pairs, bypassing explicit parameter inference.Round-specific flows are trained offline over a prescribed horizon; action flows also provide conditional log densities for uncertainty estimation.Source: §2.3 · p. 6

2.4 DECISION-FOCUSED SELECTION AMONG THOMPSON SAMPLES

• Immediate regret favors posterior expected reward, while one-step look-ahead regret scores an experiment by the subsequent recommendation it supports.Information-directed sampling combines immediate regret and information gain; an experiment with substantial immediate regret may be preferred if it improves future decisions.Source: §2.4 · pp. 6–7
• Expected information gain measures the expected reduction in uncertainty about the Thompson-sampling distribution after a hypothetical observation.Predictive observations are simulated under the current belief, and the next-round action flow supplies conditional entropy estimates.Source: §2.4 · p. 7

2.5 CANDIDATE SELECTION

• Candidate selection minimizes the chosen regret or information criterion over a finite set of approximate Thompson samples, which can exclude informative actions outside their distribution's support.The selection criteria include immediate regret, look-ahead regret, information-directed sampling, and negative information gain.Source: §2.5 · p. 8

2.6 DIRECT ACQUISITION OPTIMIZATION

• Direct acquisition optimization applies the same criteria over the full admissible action space and can identify informative experiments outside the Thompson-sampling support.The flows estimate the acquisition criterion rather than generating candidate actions.Source: §2.6 · p. 8

3.1 HOST-AWARE HETEROLOGOUS GENE EXPRESSION

• The GFP simulator models fluorescent-protein production alongside host-cell resource use, making the relationship between induction and output potentially non-monotonic.The designable input is GFP induction level, and excessive induction can overload the host and reduce final protein yield.Source: §3.1 · p. 8
• For GFP, more informative actions can improve subsequent decisions, whereas the immediate-regret action may provide little information and harm later decisions.The example history illustrates that immediate reward can favor an action close to one already observed.Source: §3.1 · p. 9
• In the GFP case, IDS outperformed vanilla Thompson sampling in both cumulative and final regret; EIG, IDS, and one-step lookahead had the best final performance among selection strategies.No statistically significant differences were detected among those three strategies; Figure 4 compares action-selection strategies, and Figure 5 reports cumulative regret.Source: §3.1 · pp. 10–11, Figure 4 · p. 11, Figure 5 · p. 11

3.2 REPRESSILATOR

• The repressilator case uses a stochastic black-box simulator to design a three-gene oscillator despite biochemical uncertainty and intrinsic reaction noise.The method requires simulated observations and rewards, not derivatives or access to the likelihood or parameter score.Source: §3.2 · p. 12, Figure 6 · p. 12
• In the repressilator, IDS had the best cumulative-regret performance, while EIG had the lowest mean final regret in the two-step setting.In the broader setting, EIG had the lowest mean final regret and other strategies performed similarly; Figure 8 and Tables 2–3 report the comparisons.Source: §3.2 · pp. 13–14, Figure 8 · p. 13, Table 2 · p. 13

3.3 DISCUSSION

• The two amortized sampling routes support rapid approximate Thompson sampling and make information-based and look-ahead criteria practical to estimate.The routes use posterior parameter inference followed by a policy, or a history-conditioned action flow; both support sequential design.Source: §3.3 · p. 14
• The parameter and action flows are trained separately, and future training objectives could encourage agreement between their representations.The action distribution should correspond to mapping posterior parameter samples through the pretrained policy.Source: §3.3 · p. 14

4 CONCLUSION

• The framework amortizes Thompson-sampling optimization before data collection, with parameter inference plus a pretrained policy and direct sampling from an action flow both effective in the case studies.It accommodates intractable likelihoods and unavailable gradients, including discrete-state Markov jump processes and a host-aware protein-expression model with over 20 coupled ordinary differential equations.Source: §4 · p. 14
• Information-gain-based selection improved final regret, while information-directed sampling improved cumulative regret in both case studies.In the first case study, directly optimizing information-gain and look-ahead criteria further improved final regret over selection, but incurred greater regret during earlier experimentation.Source: §4 · p. 15
• Selecting among Thompson-sampled candidates or optimizing decision criteria over the admissible action space lets experimenters tailor decisions to their tolerance for failures and final objective.The supported objectives include maximizing final reward or minimizing cumulative regret.Source: §4 · p. 15
• Amortization reduces computational delays between data acquisitions, while decision criteria support more efficient use of experimental budgets across a broad range of simulator models.The stated model scope includes simulators with intractable likelihoods and unavailable gradients.Source: §4 · p. 15

A.1 HOST-AWARE MODEL

• The listed reaction-network component is ribosome binding, pr + mx, for each protein x ∈ {r, t, m, q, gfp}.The passage also introduces rate definitions without specifying them in the supplied text.Source: §A.1 · pp. 17–18
• The model is adopted from Weiße et al. (2015), with a setup identical to Kobiela et al. (2026b), and its corresponding ODE system is solved using a Rosenbrock solver.The initial condition comes from a warm-up problem without GFP, with Gaussian noise added to the steady-state values.Source: §A.1 · pp. 17–18
• The uncertain parameters have priors ns ∼ U(0, 1), kb ∼ U(0, 2), and ku ∼ U(0, 2).The design parameter range is wgfp,max ∈ (1, 10000).Source: §A.1 · p. 18

A.2 REPRESSILATOR MODEL

• The repressilator model is adopted from Sequeiros et al. (2023), with a setup identical to Kobiela et al. (2026b).The section introduces reactions for each gene i ∈ {1, 2, 3}.Source: §A.2 · p. 18
• Catalyst.jl derives the Markov jump process, which is simulated using the stochastic simulation algorithm from JumpProcesses.jl.The passage cites Loman et al. (2023) for Catalyst.jl and Rackauckas & Nie (2017) for JumpProcesses.jl.Source: §A.2 · p. 18
• The uncertain parameter priors are H ∼ U(3, 7), γX ∼ U(0.8, 1.1), γm ∼ U(4, 50), and ϵ ∼ U(0.05, 0.15).The design parameter ranges are kX ∈ (100, 1000), km ∈ (3, 120), and K ∈ (10, 200).Source: §A.2 · p. 18

A.3 ROLLOUTS FOR EXAMPLE CONTEXT

• Table A1 reports regret by step and strategy for an example context, with mean ± SEM and median; Figure A1 presents its results.The supplied passages specify the table's scope and statistics but do not provide cell values or a strategy comparison.Source: Table A1 · p. 19, Figure A1 · p. 19

• Actions a1–a3 follow the specified strategy, while the final action minimizes expected immediate regret.Source: Table A2 · p. 20
• Expected regret is reported as mean ± standard error (SEM).Source: Table A2 · p. 20
• For final and cumulative regret, italics mark the best method within each selection group, bold the best overall, and highlighting marks the lowest-mean method and methods not significantly different under paired t-tests with Holm correction (α = 0.05).Source: Table A2 · p. 20

• Expected regret is estimated by Monte Carlo across four stages.Source: Table A3 · p. 21
• The table reports the standard error of the mean (SEM) alongside the estimates.Source: Table A3 · p. 21
• Median regret is also reported for the four-stage results.Source: Table A3 · p. 21