PaperlayerView exact revision on arXiv

Source-linked AI summary

Implicit Deep Adaptive Design: Policy-Based Experimental Design without Likelihoods

Desi R. Ivanova, Adam Foster, Steven Kleinegesse, Michael U. Gutmann, Tom Rainforth

arXiv:2111.02329v1stat.MLcs.AIcs.LGstat.CO

TL;DR

Adaptive BOED can require costly computation during experiments, and implicit models add intractable likelihoods. iDAD learns a policy from simulated outcomes using likelihood-free information bounds, extending policy-based design beyond conditionally independent experiments. The paper presents it as a practical real-time method for implicit models and reports that it often outperforms costly non-amortized approaches.

  • Problem

    Traditional adaptive BOED requires substantial computation during experiments, while implicit models also lack tractable likelihoods for existing policy-based methods.

    Source: §1 · p. 1, §2.1 · p. 4

  • Method

    iDAD trains a design policy and auxiliary critic on simulated histories using likelihood-free lower bounds on total information gain.

    Source: §1 · p. 2

  • Results

    iDAD is presented as a practical real-time method for adaptive BOED with implicit models and often outperforms costly non-amortized approaches.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Relaxing conditional independence extends policy-based BOED to complex settings such as time series and improves performance in conditionally independent settings.

    Source: §1 · p. 2

    iDAD requires substantial, potentially expensive training and gradients restrict the approach to continuous design settings.

    Source: §6 · p. 10

Abstract

from arXiv · show

We introduce implicit Deep Adaptive Design (iDAD), a new method for performing adaptive experiments in real-time with implicit models. iDAD amortizes the cost of Bayesian optimal experimental design (BOED) by learning a design policy network upfront, which can then be deployed quickly at the time of the experiment. The iDAD network can be trained on any model which simulates differentiable samples, unlike previous design policy work that requires a closed form likelihood and conditionally independent experiments. At deployment, iDAD allows design decisions to be made in milliseconds, in contrast to traditional BOED approaches that require heavy computation during the experiment itself. We illustrate the applicability of iDAD on a number of experiments, and show that it provides a fast and effective mechanism for performing adaptive design with implicit models.

1 Introduction

The introduction frames adaptive experiment design as a way to use collected data to choose later experiments, but traditional computation can prevent real-time decisions. iDAD learns a policy for implicit models, broadening policy-based BOED beyond likelihood-based, conditionally independent settings.

1 Introduction

  • Adaptive experiments use data already collected to guide later design choices, refining understanding of the underlying process over successive iterations.The introduction illustrates this with measurements at locations to locate a spreading contaminant.Source: §1 · p. 1
  • BOED chooses designs to maximize expected information about an unknown parameter, represented by θ, from experimental outcomes.The framework uses a prior over θ and a model relating θ and design ξ to outcome y.Source: §1 · p. 1
  • Traditional adaptive BOED fits a posterior after each iteration and selects the next design by one-step mutual information, requiring substantial computation at every step.The introduction says this approach does not lend itself to quick adaptive design.Source: §1 · p. 1
  • Likelihood-free lower bounds on total information gained train a policy network from simulated outcomes, enabling design decisions in milliseconds at deployment.Simultaneously trained auxiliary critics can also support likelihood-free posterior inference after the experiment.Source: §1 · p. 2
  • iDAD relaxes DAD’s conditional-independence requirement, allowing complex settings such as time series and improving performance in conditionally independent settings.The introduction also reports additional performance improvements from expanding the model space for policy-based BOED.Source: §1 · p. 2
  • iDAD is presented as the first practical method for real-time adaptive BOED with implicit models, often outperforming costly non-amortized approaches.Previous approaches are either too slow for real-time use with non-trivial models or require explicit likelihoods.Source: §1 · p. 2

2 Background

This section introduces BOED and the computational challenge of adaptive design, especially when likelihoods are implicit. It then describes policy-based DAD, which shifts design computation before deployment but depends on restrictive model and experiment assumptions.

2 Background

  • An implicit model can generate outcomes by simulating a transformation of noise, even when its induced likelihood density is generally intractable.The noise is independent of parameters and design, and sampling outcomes remains possible.Source: §2 · pp. 2–3
  • For implicit models, optimizing expected information gain is especially challenging because both the objective and likelihood density are intractable.The optimal design maximizes mutual information between parameters and data.Source: §2 · p. 3
  • In adaptive BOED, each design can depend on the history of earlier design-outcome pairs, while traditional methods sequentially infer a posterior and optimize a one-step objective.These computations can be costly during the experiment and often prevent real-time use, particularly for implicit models.Source: §2 · p. 3

2.1 Policy-based adaptive design with likelihoods

  • DAD learns a policy before the experiment so a forward pass can select each design, but its training requires an available likelihood and its architecture assumes conditional independence.The policy maps experimental history to the next design; these restrictions exclude implicit likelihoods and some settings such as time series.Source: §2.1 · pp. 3–4

3 Implicit Deep Adaptive Design

iDAD targets models that can be simulated and differentiated with respect to design, including models with intractable likelihoods. It builds a unified information objective for history-dependent experiments and uses variational bounds with a shared critic to train a design policy.

3 Implicit Deep Adaptive Design

  • iDAD targets models that can simulate outcomes and compute their derivative with respect to design, including implicit models with intractable likelihoods.The objective is to extend policy-based BOED beyond DAD’s likelihood and conditional-independence restrictions.Source: §3 · p. 4

3.1 Information lower bounds for policy-based experimental design without likelihoods

  • Replacing each conditional information term with a variational lower bound would require T critic networks and samples from every intermediate posterior, making the naive approach costly.The proposed bounds draw on variational mutual-information estimation while avoiding explicit likelihood evaluations.Source: §3.1 · p. 4
  • A unified total expected information gain objective remains valid when outcomes depend on history, including when the model changes between steps.Its samples can be generated by sampling the model and passing histories through the design network, although the relevant densities remain intractable for implicit models.Source: §3.1 · p. 4
  • InfoNCE and NWJ provide tractable lower bounds on the unified objective using a single auxiliary critic trained alongside the design network.The bounds adapt mutual-information estimation to policy-based design without requiring explicit likelihood evaluations.Source: §3.1 · p. 4

T (7) or LNCE

The section compares InfoNCE and NWJ bounds, describes differentiable training and deployment, and outlines architectures for the policy and critic. The method uses attention for conditionally independent histories and an LSTM when outcomes depend on history.

T (7) or LNCE

  • NWJ can be less biased and tighter with a good critic but has higher variance and is harder to estimate and optimize reliably; InfoNCE avoids self-normalization but is not generally tight at finite L.With an infinite-capacity critic and perfect optimization, only NWJ is stated to recover the true optimal policy.Source: T (7) or LNCE · p. 5
  • The trained critic can estimate a density ratio corresponding to the posterior-to-prior ratio, enabling likelihood-free posterior inference from experiment data.The paper extends this use of critics from static to adaptive policy-based design.Source: T (7) or LNCE · p. 5

3.2 Parameterization and gradient estimation

  • Gradient-based optimization uses the reparameterization trick for continuous design and observation spaces, assuming derivatives with respect to design and history are available.The method does not require the prior density or derivatives with respect to θ; simulated histories train the networks before deployment, where the policy uses a single forward pass.Source: §3.2 · pp. 5–6

3.3 Network architectures

  • The critic encodes the full history and parameter separately and scores them with a dot product, while the policy encodes history to produce the next design.The critic’s encodings are described as approximate sufficient statistics after training.Source: §3.3 · p. 6
  • For conditionally independent experiments, self-attention aggregates order-invariant history representations; when independence does not hold, an LSTM aggregates the history.The paper reports empirical gains from attention over the simpler sum-pooling approach in DAD [17].Source: §3.3 · p. 6

4 Related work

Prior adaptive-design methods include greedy posterior-based BOED and likelihood-free estimation approaches, while policy-based adaptive design had not yet been extended to implicit models.

4 Related work

  • Earlier adaptive experiments commonly use two-step greedy design, while policy-based adaptive design [17] had not been extended to implicit models.Related likelihood-free methods include variational bounds, ratio estimation, approximate Bayesian computation, and approximating the likelihood before applying likelihood-based estimators.Source: §4 · p. 7

5 Experiments

The experiments compare iDAD with heuristic, static, variational, and likelihood-based design methods, assessing information gain and deployment time. Across location finding, pharmacokinetics, and SIR problems, iDAD’s learned designs perform strongly, with qualitative examples showing adaptation to observations and model dynamics.

5 Experiments

  • The evaluation uses total expected information gain as its main metric and also measures adaptive-design deployment time.Comparisons include untrained heuristics, static BOED, an expensive sequential variational strategy, and DAD where an analytic likelihood is available; DAD is an evaluation upper-bound reference, and explicit-likelihood methods are recommended when possible.Source: §5 · p. 7

5.1 Location Finding

  • iDAD substantially outperforms all baselines on location finding, while its performance gap to likelihood-based DAD remains small as design and parameter dimensions grow.The authors attribute the variational baseline’s poor performance likely to its mean-field family’s inability to capture the highly non-Gaussian posterior; iDAD policies run in milliseconds at deployment.Source: Table 2 · p. 8, §5.1 · p. 8

5.2 Pharmacokinetic model

  • iDAD strategies perform best among methods applicable to implicit models in pharmacokinetics; InfoNCE’s gap to likelihood-based DAD is not significant at the 5% level, while NWJ’s is small but significant.The policy adapts later-patient sampling times to quickly versus slowly decaying trajectories and targets the concentration peak for another patient.Source: Figure 3 · p. 9, §5.2 · p. 9

5.3 SIR Model

  • iDAD outperforms random, equal-interval, and static designs on SIR mutual-information lower-bound estimates, although critic effectiveness may bias those estimates.For slow (R = 1.3) versus fast (R = 8.0) epidemic profiles, the learned policy selects different designs; an example critic posterior is consistent with ground truth.Source: Table 5 · p. 10, §5.3 · p. 10, Figure 5 · p. 10

6 Discussion

The discussion concludes that iDAD enables fast, adaptive design for implicit models and reports strong comparisons with likelihood-free baselines. The appendix clarifies the total-information objective and how to interpret it under deterministic and stochastic policies.

6 Discussion

  • Upfront training can be computationally expensive, and gradient use restricts iDAD to continuous design settings.Amortization across multiple experiment instances and choices of architecture and training amount can mitigate or control the cost–performance trade-off.Source: §6 · p. 10
  • iDAD performed significantly better than all likelihood-free baselines and almost matched the analogous likelihood-based adaptive approach on likelihood-available test problems.The likelihood-based approach serves as an upper bound on what might be achieved without access to the likelihood.Source: §6 · p. 10

A.1 Unified objective for non-exchangeable experiments

  • The generalized total expected information gain objective allows outcomes to depend on the preceding history, rather than requiring conditional independence across experiments.It expresses the objective using the likelihood of the full history under the policy.Source: §A.1 · p. 15

A.2 Objective function as a mutual information

  • For deterministic policies, the objective is not conventional mutual information between θ and h_T because the history lacks a Lebesgue density; with stochastic designs, it approaches that mutual information under mild assumptions.The appendix also identifies the deterministic-policy objective with mutual information between θ and y_1:T when designs are reconstructed from observations and the policy.Source: §A.2 · p. 16

A.3 NWJ and InfoNCE bounds

The appendix establishes NWJ and InfoNCE bounds for policy-based adaptive BOED, including conditions for tightness. It also explains that including designs as critic inputs makes the optimal critics independent of the policy.

A.3 NWJ and InfoNCE bounds

  • The NWJ bound holds for any critic and is tight for its optimal critic.The optimal NWJ critic is expressed using the history likelihood conditional on θ and its policy marginal.Source: §A.3 · p. 17
  • With the optimal critic, InfoNCE recovers the sPCE bound, which increases with L and becomes tight as L approaches infinity.The optimal critic has the form log p(h_T|θ, π) + c(h_T), where c depends only on history.Source: §A.3 · pp. 18–19

A.4 A note on optimal critics

  • For both bounds, including designs as explicit critic inputs makes the optimal critics independent of the policy, unlike prior approaches that omitted those inputs.The authors note this may benefit joint optimization by avoiding an implicit dependence between policy and optimal critic.Source: §A.4 · p. 19

B Theoretical Comparison and Additional Bounds

The section compares likelihood-free mutual-information bounds, showing the trade-offs between finite-sample bias, variance, and the assumptions needed for tighter bounds. It derives sLACE for adaptive designs and identifies conditions under which it can be tight.

B Theoretical Comparison and Additional Bounds

  • InfoNCE cannot exceed log(L+1), and its Monte Carlo estimator is highly biased when true mutual information exceeds that ceiling, regardless of critic quality.The cap comes from using L contrastive samples to approximate the marginal in the denominator.Source: B Theoretical Comparison and… · p. 19
  • NWJ is unbiased, but its variance grows exponentially with true mutual information, requiring exponentially many samples even with an optimal critic.The section also reports that high-confidence distribution-free lower bounds cannot be guaranteed to exceed log(N).Source: B Theoretical Comparison and… · p. 20
  • sLACE extends Foster et al.'s [17] sACE proposal-distribution bound to a sequential likelihood-free lower bound.The original sACE bound relied on an analytically available likelihood.Source: B Theoretical Comparison and… · p. 20
  • With the optimal critic, sLACE recovers sACE [17], which converges to mutual information under mild conditions.The bound is also tight as L grows with its optimal critic, and is tight for any L when q(θ; hT) is the posterior.Source: B Theoretical Comparison and… · pp. 20–21
  • The InfoNCE denominator is consistent when each term in its sum is bounded, yielding point-wise convergence to the likelihood ratio.The proof then uses bounded convergence to establish convergence of the sACE bound.Source: B Theoretical Comparison and… · p. 21
  • Neural networks parameterize the policy, critic, and proposal; a proposal closer to the posterior than the prior can benefit training.An expressive proposal and critic can make the bound tight by recovering the posterior and optimal critic.Source: B Theoretical Comparison and… · p. 22
  • Unlike InfoNCE, sLACE assumes the prior is known and its density can be evaluated.The bound also adds optimizable proposal parameters.Source: B Theoretical Comparison and… · p. 22

C.1 Permutation invariance of the critic for exchangeable experiments

For exchangeable BOED problems with conditionally independent experiments, optimal InfoNCE and NWJ critics are invariant to permutations of the history. The appendix also describes encoders and experimental setup, and explains the variational baseline’s poor location-finding performance and the limited hyperparameter search.

C.1 Permutation invariance of the critic for exchangeable experiments

  • For exchangeable BOED problems with conditionally independent experiments, the optimal critics for InfoNCE and NWJ are invariant under permutations of the history.The authors recommend permutation-invariant architectures for exchangeable models, particularly when tight bounds matter; they note that generic architectures must learn the invariance structure during optimization.Source: §C.1 · pp. 22–23

C.2 Further details on the history encoder

  • The history encoder uses self-attention followed by sum-pooling for conditionally independent experiments, and an LSTM’s last hidden state otherwise.Separate encoders serve the policy and critic, though weights outside their head layers can be shared.Source: Figure 6 · p. 23, §C.2 · p. 23

D.2 CO2 Emission Related to Experiments

  • The experiments used private infrastructure for 160 computation hours on RTX 2080 Ti or similar hardware, with estimated emissions of 17.28 kgCO2eq and 0% directly offset.Average training time per experiment, including optimization-based baselines, was 1–3 GPU hours, depending on T.Source: §D.2 · p. 24

D.3 Traditional sequential BOED with variational posterior estimator

  • The traditional sequential variational baseline optimizes a Barber–Agakov bound using a mean-field posterior approximation, then reuses its approximate posterior at each experiment stage.For location finding, the authors say this approximation likely fails to capture the complex non-Gaussian posterior, consistent with the fitted posterior after a low signal leaving many source locations plausible.Source: §D.3 · p. 24, §D.4.2 · p. 26

D.4 Location Finding

  • The location-finding setup estimates the locations of a known number of hidden sources from noisy measurements of their superposed inverse-square-law signals.The appendix assumes a standard normal prior on each location and Gaussian noise on log-total intensity.Source: §D.4 · p. 24

D.4.3 Hyperparameter selection

  • The authors did not perform an extensive hyperparameter search, focusing mainly on hidden dimension 512 and encoding dimension 64; encoding dimension 32 was marginally worse.A learning rate of 0.001 was too high, while 0.0001 yielded results very similar to the selected 0.0005.Source: §D.4.3 · p. 26

D.4.4 Further ablation studies

The further ablations examine scalability, training stability, and sensitivity to incomplete policy training. The appendix also gives the pharmacokinetic observation model and reports its hyperparameter-selection choices.

D.4.4 Further ablation studies

  • Across five additional runs, InfoNCE-trained iDAD networks show highly stable mutual-information values, while NWJ runs are more variable and have slightly lower empirical average values.The additional runs use different seeds and the same hyperparameters as the location-finding experiment.Source: §D.4.4 · p. 27
  • With only 8% of the training budget, iDAD achieves total mutual information of 7.1 versus 7.8 for the fully trained network, and with 1% it outperforms all baselines, whose best reaches 5.5.The study assesses performance as a function of training time; the corresponding ablation is reported in Table 8.Source: Table 8 · p. 28, §D.4.4 · p. 28

D.5 PK model

  • For the pharmacokinetic model, observations combine the drug concentration at sampling time with multiplicative and additive Gaussian noise.The parameters of interest are absorption rate, elimination rate, and volume; Table 9 reports bounds and deployment times, while Table 10 reports results for T = 10.Source: §D.5 · p. 28, Table 9 · p. 30

D.5.2 Hyperparameter selection

  • The pharmacokinetic networks used encoding dimensions 32 and 64, with the smaller size selected because larger networks showed no clear benefit; the NWJ learning-rate annealing factor was reduced due to high variance.The selected hidden dimension was 512, and the chosen learning rate was 0.0001.Source: §D.5.2 · p. 30

D.5.3 Further results

The appendix reports additional pharmacokinetic results and baseline stability, then details the SIR model’s differentiable SDE formulation and its training setup. For the SIR experiment, the authors pre-simulate data on a time grid because solving the SDEs is expensive.

D.5.3 Further results

  • Additional pharmacokinetic results include upper and lower information bounds and deployment times, with Table 10 covering T = 10; the static baselines are also reported to show training stability.The appendix assesses iDAD stability with five additional runs per bound using different seeds and the same experiment hyperparameters.Source: §D.5.3 · pp. 30–31

D.6 SIR Model

  • The SIR implementation uses a differentiable SDE formulation because discrete CTMC simulations do not permit gradients through sampled populations to the design variables.The model uses infected counts as observations, measurement times from 0 to 100, and pre-simulates data on a time grid to avoid repeatedly solving expensive SDEs during training.Source: §D.6 · p. 31, §D.6.1 · p. 31

D.6.1 Training details

  • For the SIR policy, encoded design–outcome histories are processed by a two-layer LSTM, and only its last hidden state is retained before the final fully connected layer.The training setup uses T = 5 experiments and 100,000 gradient steps for each bound.Source: §D.6.1 · pp. 31–32

D.6.2 Further results

For the SDE-based SIR model, expected information gain increases with the number of measurements, with the largest gain from one to two and later increases saturating. Additional runs also show training stability for iDAD and static baselines.

D.6.2 Further results

  • Expected information gain rises as measurements T increase, with the largest increase from T = 1 to T = 2 and later gains saturating.The authors relate the early jump to estimating two model parameters from measurements.Source: §D.6.2 · p. 33, Table 11 · p. 33
  • Except at T = 1, the reported expected information gains exceed those found by [29], possibly because [29] uses a noisier Poisson observation model.The authors present the observation-model difference as a possible explanation.Source: §D.6.2 · p. 33, Table 11 · p. 33
  • Five additional iDAD networks per bound, trained with different seeds and the Table 5 hyperparameters, were used to assess training stability.The reported table includes mutual-information upper and lower bounds, their mean, and standard error.Source: §D.6.2 · p. 33
  • The reported results also demonstrate training stability for the static baselines.The table reports results from repeating the same procedure for those baselines.Source: §D.6.2 · p. 33

Related papers

Loading 2111.02329v1…