Conservative Objective Models for Effective Offline Model-Based Optimization

• Offline MBO uses static data from prior experiments, but learned proxies can be fooled into directing optimization toward out-of-distribution, invalid, low-value designs.Active evaluations can be expensive or dangerous in settings such as protein optimization.Source: §1 · pp. 1–2
• COMs learn a conservative objective model by penalizing adversarially generated inputs, then optimize designs against that model with gradient ascent.The method is intended to lower-bound true objective values on out-of-distribution inputs.Source: §1 · p. 2
• On 4/7 tasks, COMs is optimal; it outperforms the best prior method by 1.3x in a high-dimensional setting and 1.16x overall.Source: §1 · p. 2

• An offline MBO algorithm receives a static dataset of inputs and their objective values while seeking an input that maximizes an unknown objective.Source: §2 · p. 2
• Typical methods fit a parametric objective model to the dataset by supervised training, then optimize the input against that model.Gradient-based optimization is one option.Source: §2 · p. 2
• When valid inputs lie on a thin high-dimensional manifold, model overestimation off-distribution can lead optimization to low-scoring or invalid designs.COMs instead train the objective model to prevent this overestimation directly.Source: §2 · p. 2

3. Conservative Objective Models for Offline Model-Based Optimization

• COMs estimate the true objective without overestimating it on out-of-distribution inputs near the training data, then optimize against that conservative estimate.Source: §3 · pp. 2–3

3.1. Learning Conservative Objective Models (COMs)

• Adversarial inputs are generated by ascending the learned objective from sampled training points; minimizing predictions on them is balanced by maximizing predictions on training data.The balance is intended to avoid systematic underestimation on in-distribution points, and the guarantee applies under assumptions.Source: §3.1 · p. 3

3.2. Optimizing a Conservative Objective Model

• The optimizer uses gradient ascent from a dataset point, with the same number of steps used to generate adversarial examples so queries remain in the conservative, lower-bound region.Source: §3.2 · p. 4

3.3. Using COMs for MBO: Additional Decisions

• A constrained objective replaces α with τ; normalized objectives let one τ work across tasks (0.5 continuous; 2.0 discrete), and the chosen T=50 horizon is reported as stable.Source: §3.3 · p. 4

3.4. Overall Algorithm and Practical Implementation

• Training alternates adversarial-sample generation and model updates, using a neural network with two 2048-unit hidden layers, leaky ReLU activations, and Adam.The final design is obtained by gradient ascent for T=50 steps.Source: §3.4 · p. 5

4. Theoretical Analysis of COMs

• The analysis assumes the input distribution used for training has support around optimizer-reached points; adding noise during ascent can satisfy this condition.In practice, μ(x_T | x_0) is a Dirac-delta distribution, while the analysis assumes positive probability across X.Source: §4 · p. 5
• Proposition 1 compares a conservative-training iterate with the corresponding iterate without conservative training, yielding a bound that includes a Lipschitz- and distance-dependent offset.The offset depends on bL and the distance between the point being evaluated and optimized solutions found from other dataset inputs.Source: §4 · pp. 5–6
• Under the analysis assumptions, conservative training makes the asymptotic model lower-bound the true function on inputs found by the optimizer when α is sufficiently large.The assumptions include Lipschitz conditions and an NTK model; the finite-step bound holds in expectation with an additional offset.Source: §4 · p. 5

Bayesian optimization

• Active Bayesian optimization methods commonly focus on low-dimensional tasks and require querying the true objective; prior work also reports their susceptibility to out-of-distribution inputs offline.Examples include Gaussian-process methods and approaches using Bayesian neural networks or ensembles (Snoek et al., 2015; Shahriari et al., 2016; Snoek et al., 2012; Angermueller et al., 2020a;b; Mirhoseini et al., 2020).Source: §5 · p. 6

Adversarial examples

• COMs is designed for offline high-dimensional inputs, where it aims to avoid out-of-distribution solutions without active querying or generative modeling.The paper connects optimizer-generated overestimation errors to adversarial inputs, while noting that smoothness-based fixes may be ineffective when the true function is non-smooth (Goodfellow et al., 2014b; Tram`er et al., 2018; Santurkar et al., 2019).Source: §5 · p. 6

Offline model-based optimization

• Offline methods often use generative models to represent valid inputs, but modeling the input manifold can be challenging and may require domain-specific tuning.COMs instead fits an approximate objective function with a regularizer and does not require a generative model; Fu & Levine (2021) avoids one but trains multiple discriminative models and quantizes y.Source: §5 · p. 6

6. Experimental Evaluation

• The evaluation covers four continuous and three discrete tasks and compares COMs with recent offline MBO methods, standard baselines, and naïve gradient ascent.Methods are evaluated using the top 128 model-selected samples and their ground-truth objective values, averaged over eight trials.Source: §6 · p. 6, §6.1 · pp. 6–7

6.1. Empirical Performance on Benchmark Tasks

• 0.985 normalized average performance puts COMs above the next-best method’s 0.852, with optimal performance on 4/7 tasks.Table 1 reports mean 100th-percentile scores and standard deviations over eight trials.Source: Table 1 · p. 7, §6.1 · p. 8
• On three of four continuous tasks, COMs attains the best results and is the only method to beat the best training point on every task.On HopperController, COMs reaches over 1.3x the best prior method; the table reports comparisons across continuous and discrete tasks.Source: Table 1 · p. 7, §6.1 · p. 8
• On discrete tasks, COMs is competitive on GFP and TF Bind8 and outperforms prior methods on UTR, reaching within one standard deviation of that task’s best method.It outperforms CbAS variants and MINs on TF Bind8 and attains almost the best performance on GFP.Source: §6.1 · p. 8

6.2. Ablation Experiments

• COMs is less sensitive than naïve gradient ascent to the number of ascent steps; the naïve model can peak at a hand-tuned step count and then degrade.Figure 3 plots ascent steps against the 100th percentile of ground-truth objective values, which is unavailable to the algorithm during optimization.Source: §6.2 · pp. 8–9, Figure 3 · p. 8
• Increasing τ improves COMs’ optimization stability, while the method remains robust to its choice; the experiments use τ = 0.5 for continuous tasks and τ = 2.0 for discrete tasks.Figure 4 varies τ on HopperController and UTR.Source: Figure 4 · p. 8, §6.2 · p. 9
• COMs remains resilient as the evaluation budget falls; on HopperController it reaches near-optimal performance with 55 evaluations, while naïve ascent needs twice that budget for its lower optimum.Figure 5 varies the number of ground-truth evaluations used to assess candidate inputs.Source: §6.2 · p. 9, Figure 5 · p. 9

7. Discussion and Conclusion

• Across four benchmark tasks, COMs matched or exceeded existing MBO algorithms, improving on the next-best method by 1.3x on the high-dimensional Hopper Controller task.The authors describe COMs as consistently high performing and more stable than prior MBO methods.Source: §7 · p. 9
• COMs’ gradient-ascent procedure could be improved with manifold modeling, which can accelerate optimization by alleviating raw-input traversal.Source: §7 · p. 9
• Learned objective models are prone to overfitting, especially with limited data, a concern the authors identify as relevant to practical MBO problems.They suggest understanding overfitting mechanisms could broaden COMs’ applicability to small-dataset settings.Source: §7 · p. 9

Appendices

• The appendix reports a 50th-percentile evaluation of COMs against other methods.Source: A. Method Details · p. 13
• It discusses additional COM details, including hyperparameters.Source: A. Method Details · p. 13
• It also discusses how the benchmarking tasks are curated.Source: A. Method Details · p. 13

A.1. Additional Results

• COMs was optimal on 4/7 tasks and achieved the greatest normalized average performance, 0.590, in the mean 50th-percentile evaluation.Table 2 reports mean 50th-percentile scores and standard deviations over eight trials across seven tasks.Source: §A.1 · p. 13, Table 2 · p. 13

A.2. Implementation details

• Inputs and outputs are normalized so COMs can use a uniform set of hyperparameters across tasks with differing ranges and modalities.Source: §A.2 · pp. 13–14
• COMs keeps hyperparameters constant across discrete tasks and across continuous tasks; d denotes the cardinality of a training design x.Source: Table 3 · p. 14
• The implementation follows the Gradient ascent baseline from Trabucco et al. (2021).Source: §A.2 · p. 14
• The normalization subtracts each sample mean and divides by its sample standard deviation, except when that would divide by zero.The procedure is applied to both inputs and objective values.Source: §A.2 · p. 14

A.3. Benchmarking Details

• Design-Bench task identifiers and keyword arguments specify inputs to the task-building function for reproducible benchmark construction.The dataset name and desired oracle function both need to be specified.Source: §A.3 · p. 14, Table 4 · p. 14
• Approximate oracles can disagree with dataset ground-truth labels; for UTR, the authors relabeled labels using the CNN oracle predictions.The authors report UTR as particularly susceptible to this discrepancy.Source: §A.3 · p. 14

B. Proof of Theorem 1

• The proof derives the training-loss gradient, uses an approximate linearization, and shows each gradient-descent step reduces expected predictions relative to squared-error-only training.The proof treats the objective as deterministic for simplicity and states it extends to nondeterministic values with an additional integral.Source: B. Proof of Theorem 1 · pp. 14–15
• Under the theorem’s assumptions, sufficiently large α makes the asymptotic conservative model’s expected value on optimizer-found inputs lower-bound the true objective.The proof assumes small-learning-rate NTK behavior, a semi-positive-definite kernel, and a model Lipschitz constant sufficiently larger than the true function’s.Source: B. Proof of Theorem 1 · pp. 15–16

C. Network Details

• Experiments use a shared three-layer neural network with two 2048-unit hidden layers, Leaky ReLU leak 0.3, and a scalar regression output.The network is trained to minimize mean squared error using Adam’s default parameters.Source: C. Network Details · p. 16

D. Data Collection; E. Oracle Functions

• The task suite documents each dataset’s collection and preprocessing, while oracle functions generally use larger hidden datasets or exact evaluations.The benchmark description covers six tasks spanning DNA and protein design, superconductors, controller weights, and robot morphology.Source: D. Data Collection · p. 16, E. Oracle Functions · p. 18

D.1. TF Bind 8; E.1. TF Bind 8

• TF Bind 8 uses all 65,792 length-eight DNA sequences, but exposes only the bottom 50% as 32,898 training examples; its oracle is a score lookup table.The task selects binding scores for transcription factor SIX6 REF R1.Source: §D.1 · p. 16, §E.1 · p. 18

D.2. GFP; E.2. GFP

• GFP inputs encode 237-amino-acid proteins; MBO sees 5,000 mid-range examples, while a Transformer oracle is fit to the full 56,086-protein dataset.The oracle’s held-out Spearman rank correlation is 0.8497.Source: §D.2 · p. 16, §E.2 · p. 18

D.3. UTR; E.3. UTR

• UTR sequences are length 50, with the top 280,000 by total reads retained and the top half by ribosome load excluded from MBO training.A residual CNN oracle is fit to the full hidden dataset and has held-out Spearman rank correlation 0.8617.Source: §D.3 · p. 17, §E.3 · p. 18

D.4. Superconductor; E.4. Superconductor

• Superconductors use 86-component chemical-formula vectors; MBO training uses 17,010 of 21,263 samples, and a random-forest oracle is trained on the full hidden dataset.The representation was chosen to support recovery of physically realizable materials; held-out Spearman rank correlation is 0.9155.Source: §D.4 · p. 17, §E.4 · p. 18

D.5. Hopper Controller; E.5. HopperController

• Hopper Controller optimizes 5,126 policy weights from 3,200 collected examples; its exact oracle evaluates a 1,000-step Hopper-v2 rollout.Data come from 32 PPO trials, with policy weights saved every 10,000 environment steps.Source: §D.5 · p. 17, §E.5 · p. 18

D.6. Ant & D’Kitty Morphology; E.6. Ant & D’Kitty Morphology

• Ant and D’Kitty tasks optimize 60- and 56-dimensional morphologies using pretrained morphology-conditioned policies; each MBO dataset has 10,004 examples.Their exact MuJoCo oracles evaluate rollouts limited to 100 steps; the full collected datasets contain 25,009 morphologies each.Source: §D.6 · pp. 17–18, §E.6 · pp. 18–19