Designing Experiments for External Decision Makers

1 INTRODUCTION

• Standard BED assumes the experimenter chooses the downstream action, making more information beneficial in expectation under its defining objectives.Existing approaches for external rules address only narrow cases and use static designs or bespoke derivations.Source: §1 · pp. 1–2
• Usage-aware experimental design minimises the experimenter’s expected loss when downstream rules, rather than the experimenter’s Bayes act, choose actions.Rules may be non-Bayesian or black-box; uncertainty over rules can be handled by averaging usage risk or minimising its worst case.Source: §1 · p. 2
• For rules other than the experimenter’s Bayes act, informative data can steer decisions toward worse outcomes, so no uncertainty-based criterion generally captures usage risk.The framework strictly generalises standard BED.Source: §1 · p. 2
• Usage-aware designs often substantially outperform designs that ignore the downstream rule across clinical trials, consumer persuasion, image acquisition, and ranking and selection.They also collect qualitatively different data.Source: §1 · p. 2

2 BACKGROUND

• A design’s Bayes risk is the expected loss of the Bayes act under the predictive distribution of possible outcomes, and classical BED minimises it.The Bayes act minimises posterior expected loss.Source: §2 · p. 2
• Expected posterior uncertainty is equivalent to Bayes risk when the uncertainty measure is a generalised entropy derived from a proper scoring rule.For Shannon or differential entropy, minimising expected posterior uncertainty is equivalent to maximising expected information gain.Source: §2 · p. 2
• In sequential BED, a policy selects each experiment using the history of earlier designs and outcomes, making Bayes risk a function of the policy.The policy affects risk through the distribution of histories.Source: §2 · p. 3

3 EXPERIMENTAL DESIGN FOR EXTERNAL DECISION MAKERS

• A downstream decision maker may be external or may represent a fixed rule the experimenter must follow, and maps the completed data history to an action.The experimenter’s loss is evaluated on that action under their own beliefs and preferences.Source: §3 · p. 3
• Standard BED is recovered when the downstream decision maker applies the experimenter’s own Bayes act; otherwise the design is optimal conditional on the rule actually applied.The rule need not be Bayesian.Source: §3 · p. 3
• When rule parameters are uncertain and unobserved during data collection, the framework models them jointly with the unknown state and data history.The parameters can represent discrete choices among procedures.Source: §3 · p. 3
• For multiple possible decision makers, average usage risk can weight each rule by its prevalence or by how much the experimenter values its actions.A worst-case objective instead minimises the largest usage risk over plausible rules.Source: §3 · pp. 3–4
• Randomised design policies can strictly reduce worst-case usage risk, although randomisation cannot improve the average-risk objectives described here.The worst-case criterion is a robust decision alternative rather than an expected loss.Source: §3 · p. 4
• Policies are typically trained by stochastic gradient descent on simulated loss estimates, using rule evaluations and gradient estimators suited to differentiable or black-box rules.The policy and simulator’s reparameterisability and the rule’s differentiability determine which estimator can be used.Source: §3 · p. 4

4.1 Design-Dependent Incompatibility

• The incompatibility cost is the expected excess loss from the fixed rule acting instead of the Bayes act, and it bounds the price of ignoring that rule.The cost is design-dependent; under squared error it is expected squared distance from the posterior mean, and under log loss it is expected KL divergence from the posterior to the rule’s reported distribution.Source: §4.1 · p. 4

4.2 When Does More Information Help?

• More informative data help if and only if the fixed rule’s posterior risk is concave; otherwise they can hurt.The stated result assumes a finite state space, a full-support prior, and a rule that acts as a function of the posterior.Source: §4.2 · p. 5
• When posterior risk is nonconcave, usage risk cannot induce the same experiment ordering as standard BED under any loss.The result also excludes expected posterior uncertainty criteria, including expected information gain; usage-aware design departs from BED exactly when the rule’s posterior risk is not a measure of uncertainty.Source: §4.2 · p. 5
• When the data carry no information about which rule will act, averaging over rules can make posterior risk concave and restore the no-harm guarantee on average.The guarantee holds when the mixture of the rules’ posterior risks is concave.Source: §4.2 · p. 5

5 RELATED WORK

• Earlier work on second Bayesians and multiple Bayesians studies specified decision makers, with multiple-rule work minimizing weighted average risks in static designs.Examples include differing priors or losses (Jackson et al., 1980; Lindley and Singpurwalla, 1991; Etzioni and Kadane, 1993; De Santis and Gubbiotti, 2017, 2021), differing fitted models, and multiple Bayesians.Source: §5 · p. 5
• Bayesian persuasion work characterizes a single static optimal signal for one known receiver and tractable models, whereas this framework learns adaptive policies by simulation for uncertain rules.The cited work also considers receivers with different worldviews, non-Bayesian updating, or frequentist tests (Galperti, 2019; de Clippel and Zhang, 2022; Ball and Espín-Sánchez, 2021).Source: §5 · p. 6
• Clinical-trial design for prescribed tests includes static assurance and bespoke adaptive-trial or stopping solutions; the paper says its method applies to these problems unchanged.Examples include assurance (Spiegelhalter and Freedman, 1986; O’Hagan et al., 2005; De Santis et al., 2025), adaptive trials (Ondra et al., 2019), and genomics stopping (Rossell and Müller, 2013).Source: §5 · p. 6

6.1 Adaptive Clinical Trials

• Under Bonferroni, the learned policy recovers 99.7% of the oracle DP solution’s expected utility and improves on the best single-stage design by 10%.Under Simes, where no DP solution is available, the policy trained for that test matches or exceeds every other design in the table.Source: §6.1 · p. 6, Table 1 · p. 6

6.2 Persuading a Sceptical Consumer

• The usage-aware policy’s usage risk is 36% below the best baseline and about half that of random inspection.It sends 96% of its first five inspections to the four zones the consumer suspects and stops before the fifteenth inspection on only 43% of lots.Source: §6.2 · p. 7, Figure 1 · p. 7

6.3 High-Dimensional Image Acquisition

• The usage-aware policy is most accurate at every precision budget, with its largest advantage when precision is scarce.Transferred and Action-BED policies perform worse; as the budget grows, all policies measure most of the image well and their gaps narrow.Source: Figure 2 · p. 7, §6.3 · p. 8

6.4 Bayesian Ranking and Selection

• As the decision maker grows more cautious, KG’s usage risk rises faster; EIG maximisation and random sampling do worse still.At κ = 3, the usage-aware policy accepts somewhat higher Bayes risk to cut incompatibility cost by two thirds. With two decision makers, average and minimax policies beat their respective specialist baselines, most when few candidates are shared.Source: §6.4 · pp. 8–9, Figure 3 · p. 8

• The framework targets external decision makers whose rules the experimenter cannot change and may not know, including non-Bayesian or black-box rules.It learns adaptive policies by simulation using evaluations of the rule, and recovers standard BED when the rule is the experimenter’s Bayes act.Source: §7 · p. 9

A.1 Proof of Proposition 2

• For sufficiency, garbling makes the less informative posterior a conditional mean of the more informative posterior, so concavity and Jensen’s inequality imply the risk ordering.The sufficiency argument does not require full support.Source: §A.1 · p. 12
• For necessity, the proof uses a three-outcome experiment whose refinement splits a posterior into two beliefs, then merges those outcomes in a garbled experiment.This construction makes the usage-risk inequality fail when posterior risk is not concave; full support is used only for necessity.Source: §A.1 · pp. 12–13

A.2 Proof of Corollary 1

• No Bayes risk induces the same ordering as usage risk on any class of experiments containing the constructed pair.The proof compares the posterior mixture with its split into component beliefs, yielding opposite risk orderings.Source: §A.2 · p. 13

B.1 Adaptive Clinical Trials

• The trial study uses a two-stage targeted-therapy problem with a prespecified regulator test and societal utility; its dynamic-programming oracle requires closed-form rejection probabilities.That requirement restricts the oracle to Bonferroni, while the simulation-based method can also use Simes.Source: §B.1 · p. 14
• The learned trial policy trains one interim network across 100 first stages, then selects the best first stage by simulation; every trained policy selects the DP’s (92.8, 54.9).The selected stage is judged by the trial it leads to, and the DP and learned method choose from the same 100 options.Source: §B.1 · p. 15
• 87.5 million euros for prior approval versus 75.5 million for the best trial under the Bayes-act assumption; either actual test would approve no one without a trial.The comparison is limited to single-stage designs under that assumption.Source: §B.1 · p. 16
• For clinical-trial comparisons, all designs are evaluated under both tests on shared test trials, with paired differences and seed variation reported.The DP takes 54 minutes; each policy trains in 16 to 26 seconds on the same CPU.Source: §B.1 · p. 16

B.2 Persuading a Sceptical Consumer

• The consumer study turns acceptance sampling into sequential inspection of a 16-zone ring, where defects cluster around an unknown weak region and inspections destroy units.The manufacturer chooses inspection zones and when to stop; the consumer uses a more sceptical prior and accepts only above posterior probability 0.8.Source: §B.2 · pp. 16–17
• The consumer experiment’s usage-aware policy chooses inspection zones with an actor network and uses a critic-based rule to stop when stopping’s reward meets the estimated value of continuing.During training it continues with probability 0.05 after the stop condition, so the critic can learn continuation value in those states.Source: §B.2 · p. 17
• Consumer-study baselines include Action-BED against the manufacturer’s Bayes risk, RL-BOED against an information-gain lower bound, and random inspection; the latter two use all 30 inspections.The study also reports a full-budget Action-BED ablation.Source: §B.2 · p. 17
• Each learned consumer policy uses five seeds and is evaluated on 10,000 new lots per seed, with stopping applied without exploration before the consumer decides.Table 2 reports mean and standard deviation across seeds; Figure 1 pools zone frequencies and stopping curves across evaluation episodes.Source: §B.2 · p. 17

B.3 High-Dimensional Image Acquisition

• Each policy allocates a fixed per-round precision budget across 196 image blocks over four rounds, with no choice over total spending or stopping.The four total budgets are 784, 1,568, 3,136, and 6,272.Source: §B.3 · p. 18
• The two frozen downstream classifiers differ only in their historical sensing distributions: one is biased toward the top of digits and the other toward the bottom.They share initialization, training images, placements, sensor noise, and budgets; their sensing histories are vertical reflections.Source: §B.3 · pp. 17–18
• The usage-aware policy is trained separately for each frozen classifier and total budget, while the transferred baseline uses the adaptive policy trained for the other classifier.The static policy learns observation-independent allocations; Action-BED is adaptive but agnostic to the downstream classifier.Source: §B.3 · pp. 19–20
• 92.6%, 96.0%, 97.3% and 97.8% are Action-BED’s accuracies at the four budgets when evaluated by its own classifier, exceeding every policy under the downstream classifiers in Figure 2.The authors attribute its weaker downstream accuracy to mismatch with the downstream rule rather than an uninformative design.Source: §B.3 · p. 20
• Evaluation averages accuracy over three independently trained classifier pairs and three policy seeds per method and target, using 10,000 test images and three paired placement-and-noise replicates.Each pair shares placements and noise across compared models.Source: §B.3 · pp. 18, 20

B.4 Bayesian Ranking and Selection

• The controlled study varies the mismatch between a downstream rule and the Bayes act, then considers a single experiment for two decision makers with different rules and candidate options.The conjugate Gaussian setting allows the rule to move continuously away from the Bayes act, and knowledge gradient provides a strong baseline.Source: §B.4 · p. 20
• The usage-aware knowledge-gradient policy did not match the usage-aware KG(*) policy in this conjugate Gaussian model, so the study uses KG(*) while holding the design method fixed across rules.The authors report that neither REINFORCE nor PPO matched it.Source: §B.4 · p. 21
• For two decision makers, the average policy weights their terminal values equally, while the minimax policy selects a weight to maximise the lower of their expected deployed utilities on validation episodes.The study also compares minimax against the best episode-level randomisation between the specialist policies.Source: §B.4 · pp. 21–22

C DETAILS ON POLICY LEARNING

• Pathwise gradients apply when designs, simulator, and loss-through-rule are differentiable; otherwise, a discrete design or black-box or nondifferentiable rule calls for stochastic policy gradients.The pathwise estimator backpropagates through the rule and simulator; the score-function estimator only requires evaluating them.Source: C DETAILS ON POLICY LEARNING · p. 22
• The policy-learning formulation treats history as state, the next design as action, simulated experiments as transitions, and terminal negative loss as reward.PPO adds a learned critic baseline and a clipped surrogate objective; the paper notes that learned policies did not match usage-aware KG(*) in Section 6.4.Source: C DETAILS ON POLICY LEARNING · p. 23

C.1 Uncertain Downstream Rules

• The average and minimax objectives extend the same simulation-based policy-learning recipe: draw rule parameters jointly with the state for averaging, or combine rule-specific risks for minimax.For finite or discretised rule sets, a log-sum-exp upper bound yields a weighted gradient estimate that is biased at finite simulation sample size but consistent as sample size grows.Source: §C.1 · p. 23

Access to the downstream rule

• Usage-aware design requires access to an evaluable downstream rule or plausible rule set, and its performance depends on how well that set is specified.A policy tailored to one rule can perform poorly under another; minimax protection covers only rules in the chosen set Ψ.Source: §D.1 · p. 23

Scope of the theory

• The theory assumes finite Z, a full-support prior, and rules that use data only through the experimenter’s posterior.It does not cover rules using information discarded by the posterior, such as classifiers applied directly to raw measurements; extending the characterisation to infinite Z is not straightforward.Source: §D.1 · p. 23

Optimisation

• Discontinuous rules often preclude pathwise gradients, while simulation can be costly and neural policies did not match an adapted knowledge-gradient policy.Training may therefore require higher-variance score-function estimators.Source: §D.1 · p. 23

Influence and the Bayes act

• A DM sharing the likelihood retains a correct posterior regardless of the design policy, and their posterior averages back to their prior.Consequently, a DM taking their own Bayes act cannot be harmed in expectation by the data.Source: §D.2 · p. 24
• The guarantee compares with having no data, so experimenters may still choose informative experiments that suit them rather than the DM.A floor on the DM’s expected welfare is suggested as a possible regulatory response.Source: §D.2 · p. 24
• Under non-Bayes rules, more data can harm a DM, and the paper characterises vulnerability through concavity of the rule’s posterior risk.The guarantee also fails if the experimenter withholds data, because the DM may then use an incorrect posterior.Source: §D.2 · p. 24