PaperlayerView exact revision on arXiv

Source-linked AI summary

Conformal Policy Control

Drew Prinster, Clara Fannjiang, Ji Won Park, Kyunghyun Cho, Anqi Liu, Suchi Saria, Samuel Stanton

arXiv:2603.02196v4cs.AIcs.LGmath.STstat.MLstat.TH

TL;DR

Agents need to improve behavior without violating safety constraints, but safe defaults can be overly conservative and choosing among apparently safe candidates can bias risk estimates. CPC calibrates policy interpolation using data from a safe reference policy, and experiments across three tasks report risk control alongside performance benefits. Its guarantees are marginal and depend on context-distribution stability.

  • Problem

    Agents may need to improve an untested policy while controlling risk, but a safe default can be far from optimal and selecting candidates by estimated risk can bias that estimate.

    Source: §1 · p. 1

  • Method

    CPC calibrates a likelihood-ratio threshold on safe-policy data to interpolate between a safe reference policy and an optimized policy at the user’s risk tolerance.

    Source: §1 · p. 1

  • Results

    Across three tasks, gCRC controls medical-answer false discovery rate while improving claim recall, CPC provides finite-sample guarantees under feedback-loop covariate shift, and moderate control can improve biomolecular optimization efficiency.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Safe-policy data can support risk-controlled exploration, and moderate risk control can reduce wasted evaluations on infeasible actions in biomolecular optimization.

    Source: §1 · pp. 1–2

    Guarantees are marginal over contexts and calibration draws, and context-distribution shifts may require recalibration or tolerating a risk-control gap.

    Source: §6 · p. 9

Abstract

from arXiv · show

An agent must try new behaviors to explore and improve. In high-stakes environments, an agent that violates safety constraints may cause harm and must be taken offline, curtailing any future interaction. Imitating old behavior is safe, but excessive conservatism discourages exploration. How much behavior change is too much? We show how to use any safe reference policy as a probabilistic regulator for any optimized but untested policy. Conformal calibration on data from the safe policy determines how aggressively the new policy can act, while provably enforcing the user's declared risk tolerance. Unlike conservative optimization methods, we do not assume the user has identified the correct model class nor tuned any hyperparameters. Unlike previous conformal methods, our theory provides finite-sample guarantees even for non-monotonic bounded loss functions, and it introduces a new policy control setting. Our experiments on applications ranging from natural language question answering to biomolecular engineering show that safe exploration is not only possible from the first moment of deployment, but can also improve performance.

1. Introduction

Conformal policy control (CPC) uses data from a known-safe policy to regulate an optimized, untested policy according to a user-declared risk tolerance. It combines policy interpolation with conformal calibration to support exploration while controlling risk.

1. Introduction

  • The setting balances exploration against safety: unsafe behavior can cause harm, while a known-safe default policy may be far from optimal.In consequential settings, collecting data from an unsafe behavior policy may be unacceptable, and deriving the balance from assumptions reintroduces dependence on the practitioner’s understanding of the problem.Source: §1 · p. 1
  • Choosing the most aggressive candidate that merely appears safe can bias risk estimates, because selection favors candidates whose estimated risk is too low.The agent can reweight safe-policy observations by each candidate-to-safe likelihood ratio, since it knows how candidate policies distribute their actions.Source: §1 · p. 1
  • CPC interpolates between a safe reference policy and an optimized policy, using calibration data from the safe policy to determine how aggressively the new policy can act.The user supplies a risk tolerance; rejection sampling can realize the regulated policy at deployment.Source: §1 · pp. 1–2
  • The generalized conformal risk-control method (gCRC) provides finite-sample guarantees for bounded, non-monotonic losses, where standard CRC’s monotonicity condition does not apply.CPC’s control parameter governs the policy rather than the loss.Source: §1 · p. 2
  • Across three tasks, CPC and gCRC control risk while supporting stronger performance: gCRC improves medical-answer claim recall, and moderate control can improve biomolecular optimization efficiency.In constrained active learning, CPC provides finite-sample guarantees under feedback-loop covariate shifts where prior approaches offered only asymptotic control.Source: §1 · p. 2

2. Preliminaries and Background

The section formalizes policy improvement under a user-specified risk tolerance and motivates selecting a policy from safe-policy data. It contrasts standard conformal risk control, which relies on loss monotonicity, with policy control where that condition may fail.

2. Preliminaries and Background

  • A policy maps contexts to action distributions, and the paper describes reward and loss as measures of objective alignment and constraint violation for context-action pairs.Contexts may be prompts, and the setting includes explicit conditional policies such as autoregressive transformers.Source: §2 · p. 2

2.1. The Safe Policy Improvement Problem

  • The safe policy improvement problem seeks a policy that maximizes expected reward while keeping expected loss within the user’s risk tolerance.A safe reference policy is a practical starting point, but users typically must otherwise tune divergence limits or penalty weights to reach their desired risk level.Source: §2.1 · p. 3
  • Importance weighting estimates a candidate policy’s expected loss by reweighting safe-policy observations with the candidate-to-safe likelihood ratio.Bounding this ratio at β also constrains policy divergence, including KL divergence and other f-divergences.Source: §2.1 · p. 3, §2.2 · p. 3

2.2. Conformal Risk Control

  • Standard conformal risk control selects a control parameter from exchangeable calibration losses and provides a finite-sample risk guarantee when loss decreases monotonically with that parameter.Its search stops at the smallest parameter whose conservatively adjusted empirical risk is at most the target α.Source: §2.2 · p. 3
  • Policy control places the parameter on the action distribution rather than the loss, so expected loss need not vary monotonically with the parameter.For example, an environment’s feasibility set is fixed; changing the policy can change action probabilities, not the feasibility of a given action.Source: §2.2 · p. 3

3. Abbreviated Related Work

Prior safe optimization methods often regulate risk through policy divergence, while earlier conformal approaches chiefly provide guarantees for fixed policies or restricted losses. This paper’s related-work discussion situates CPC and generalized CRC as addressing policy selection and non-monotonic-loss guarantees.

3. Abbreviated Related Work

  • Conservative policy-optimization methods bound risk through divergence from a safe reference, but likelihood-ratio thresholds and related control choices have often been treated as hyperparameters.The cited approaches include trust-region, KL-penalized, conservative offline-RL, and uncertainty-penalized optimization methods.Source: §3 · p. 4
  • Earlier conformal optimization work was limited to miscoverage, faced an implicit-policy fixed-point problem, and did not fully address multi-step feedback covariate shift [Stanton et al., 2023].Other methods established coverage for fixed policies, but selecting a policy after calibration can introduce dependence that nullifies those guarantees.Source: §3 · p. 4
  • On synthetic non-monotonic losses, gCRC controls risk across target levels while standard CRC does not; the example trajectory shows CRC underestimating risk while gCRC searches from safe to aggressive values.The figure compares standard CRC (orange) with generalized CRC (blue).Source: Figure 2 · p. 4
  • The paper extends conformal risk control to finite-sample guarantees for non-monotonic losses and applies techniques for multi-round model-based optimization.The cited prior CRC work left finite-sample guarantees for non-monotonic losses open [Angelopoulos et al., 2024].Source: §3 · p. 4
  • Figure 3 illustrates rejection sampling as β-controlled interpolation: small β nearly recovers the safe policy, while large β approaches the optimized policy with fewer rejections.Accepted samples are shown in blue; rejected samples come from the proposal distribution.Source: Figure 3 · p. 5, §3 · p. 5

4. Methods and Theory

Generalized conformal risk control extends finite-sample risk control to non-monotonic losses, while conformal policy control calibrates how far an optimized policy may move from a safe one. Both select a risk-controlled option from calibration data, with guarantees relying on stated regularity and stability conditions.

4.1. Extending Conformal Risk Control to Non-Monotonic Losses

  • gCRC selects the smallest hyperparameter whose empirical risk remains controlled at α for every larger hyperparameter, then conservatively adjusts for the unknown test loss.This all-larger-values condition is the key modification to standard CRC; gCRC reduces to CRC when losses are monotonic.Source: §4.1 · p. 5
  • Finite-sample guarantees for non-monotonic losses hold under Lipschitz losses and replace-one stability of the empirical procedure.The method relates the empirical procedure to an oracle rather than relying on loss monotonicity.Source: §4.1 · p. 5

4.2. Conformal Policy Control: Selecting a Risk-Controlling Policy

  • CPC assumes a safe policy with risk at most α and access to optimized policies; it imposes no other assumptions on how those policies are trained.The policies must be mutually absolutely continuous.Source: §4.2 · p. 6
  • CPC clips the optimized-to-safe likelihood ratio at β, then calibrates the largest β whose conservatively adjusted weighted empirical risk remains controlled.Increasing β permits more aggressive deviation from the safe policy; calibration searches from safer to more aggressive settings.Source: §4.2 · p. 6
  • The CPC guarantee requires bounded losses, policy absolute continuity, and relative replace-one stability; its smoothness condition applies to the agent-controlled policy rather than the unknown loss.The likelihood-ratio control parameter makes the required policy smoothness enforceable.Source: §4.2 · p. 7

5. Experiments

Across medical QA, constrained active learning, and black-box sequence optimization, conformal methods control risk while maintaining or improving the task objective. The experiments test non-monotonic factuality loss and policy shifts in data selection and sequence generation.

5. Experiments

  • The experiments apply gCRC to medical QA, CPC to active learning under feedback-loop shifts, and CPC to language-model black-box sequence optimization.Appendix B adds a constrained Bayesian optimization comparison with standard conservative optimization.Source: §5 · p. 7

5.1. Factual Medical Question-Answering

  • gCRC controls medical QA false discovery rate at or below target α across risk levels, with recall at least as high—and often much higher—than the baselines.The comparison includes Learn Then Test and CRC applied to monotonized losses.Source: §5.1 · p. 8

5.2. Constrained Active Learning

  • CPC controls constraint-violation risk at target α while reducing test MSE across three active-learning datasets.In some cases, the risk-controlled policy achieves lower test MSE than the unconstrained policy.Source: §5.2 · p. 8, Figure 5 · p. 8

5.3. Constrained Black-Box Optimization

  • CPC controls infeasible-sequence risk while improving objective value over time; at α > 0.6, controlled policies can slightly outperform uncontrolled ones.The authors attribute the improvement likely to fewer evaluations wasted on infeasible actions.Source: §5.3 · p. 9, Figure 6 · p. 9

6. Discussion

Conformal policy control calibrates how far a policy can deviate from a safe reference, so the selected policy itself receives a risk guarantee. The approach requires a safe policy and declared risk tolerance, while its guarantees are marginal and depend on context-distribution stability.

Discussion

  • CPC calibrates a likelihood-ratio threshold on safe-policy data, making policy selection itself the calibrated act and bounding deviation for the chosen policy.The guarantee applies to the selected policy, not merely to each candidate considered separately.Source: §6 · p. 9
  • CPC uses conformal methods prescriptively to search over distributions satisfying a finite-sample risk guarantee, a framing applicable to other risk-constrained data-generating choices.The paper names experimental design, active data collection, and generative-model deployment as possible settings.Source: §6 · p. 9
  • The guarantees control risk on average over contexts and calibration-data draws, so they are less informative for individual decisions and depend on context-distribution stability.Context shifts may require recalibration or tolerating a risk-control gap; stronger conditional guarantees require additional assumptions or more conservative bounds.Source: §6 · p. 9
  • CPC requires an acceptable-risk baseline, smooth policies enforced through a likelihood-ratio bound, and a directly declared risk tolerance rather than an indirect penalty weight.The discussion frames safety and exploration as potentially complementary when balanced using what is known.Source: §6 · p. 9

Impact Statement

The impact statement contrasts reactive deployment practices with specifying acceptable risk before deployment. CPC aims to make performance tuning occur within safety constraints, with broader adoption effects presented conditionally.

Impact Statement

  • The paper describes a common deployment pattern as training and tuning until performance seems acceptable, then reacting to failures after harm is observed.It says principled alternatives have been lacking, rather than claiming practitioners prefer this pattern.Source: Impact Statement · p. 10
  • CPC lets practitioners specify acceptable risk before deployment and obtain guarantees that those levels will be respected.It reframes tuning as improving performance within safety constraints rather than searching for an unspecified feasible configuration.Source: Impact Statement · p. 10
  • If such methods become widespread, formal guarantees may make some high-stakes domains more accessible despite liability or regulatory barriers.The paper names clinical decision support, autonomous systems, and financial services as examples.Source: Impact Statement · p. 10

Supplementary Material

The supplementary material situates CPC among conservative optimization approaches that constrain deviation from trusted policies or data. It motivates using safe-policy data for calibration because policy shifts are agent-chosen and their likelihood ratios are known.

Supplementary Material

  • Selection among uncertain alternatives can favor actions with overestimated values, and aggressive policy optimization can shift data away from regions where estimates are reliable.The supplementary discussion describes this as the optimizer’s curse and notes that sequential distribution shifts can compound estimation error.Source: §A.1 · p. 16
  • Conservative control methods constrain optimized policies to remain close to reference distributions or trusted regions, but their constraint strength is governed by a scalar hyperparameter.The listed approaches include KL-regularized control, trust-region methods, conservative offline reinforcement learning, and safe Bayesian optimization.Source: §A.1 · p. 16
  • Choosing that hyperparameter may require costly on-policy evaluations of candidate settings, because risk depends on the distribution each setting induces.The discussion also notes added feedback-loop dependencies from this calibration process.Source: §A.1 · p. 17
  • Because the agent chooses its policy and knows candidate-to-baseline likelihood ratios, the hyperparameter search can use off-policy safe-policy data when calibrated carefully.The supplementary discussion identifies conformal techniques as a way to perform that careful calibration.Source: §A.1 · p. 17

A.2. Conformal Methods

The related-work discussion distinguishes descriptive conformal prediction from methods that select a risk-controlling policy. It places CPC among work on distribution shift, offline and active decision-making, risk control, and Seldonian algorithms.

Conformal prediction under distribution shift in high-dimensional spaces

  • In high-dimensional action spaces, evaluating density ratios can be computationally difficult; CPC instead makes a clipped policy density-ratio bound the tunable control parameter.The bound avoids estimating the density ratio between calibration and test policies.Source: Conformal prediction under… · pp. 17–18

Conformal methods for decision-making in the offline setting

  • Offline conformal decision methods leave open the practical problem of controlling how prediction error affects the downstream policy, rather than prediction sets alone.The cited discussion characterizes much prior decision-making work as offline, with training and calibration independent of test-time decisions.Source: Conformal methods for decision-making… · p. 18

Conformal methods for decision-making in the active setting

  • Unlike active conformal methods that adapt prediction sets for a fixed policy, CPC searches for a risk-controlling policy and avoids an extra data split through reweighting.The discussion also distinguishes CPC from approaches bottlenecked by density-ratio estimation or focused on miscoverage.Source: Conformal methods for decision-making… · pp. 18–19

Conformal selection and high-probability risk control

  • Conformal selection methods control false discovery among a pool of candidates, while CPC focuses on selecting a generative policy with an expected risk guarantee.The discussion contrasts these expected-value guarantees with high-probability risk-control approaches and emphasizes policy control as the setting addressed.Source: Conformal selection and… · p. 19

A.3. Seldonian Algorithms

  • Seldonian algorithms certify returned solutions with high probability, whereas CPC provides marginal expected-risk guarantees and can regulate an existing model trained by another algorithm.The discussion says neither guarantee type strictly dominates: high-probability bounds are more conservative, while expected-value bounds permit occasional violations.Source: §A.3 · p. 19

B. Comparison to Conservative Optimization on Constrained Bayesian Optimization Task

The comparison tests CPC as a regulator for constrained Bayesian optimization across acquisition temperatures. CPC keeps constraint violations below the declared target while maintaining comparable or improved reward, whereas the baseline remains above the target and may require further tuning.

B. Comparison to Conservative Optimization on Constrained Bayesian Optimization Task

  • Standard conservative optimization methods often assume a particular model class and cannot directly translate a user's risk tolerance into an operating parameter, making hyperparameter tuning necessary.The comparison frames CPC as complementary: any exploration policy can serve as its optimized policy.Source: B. Comparison to Conservative… · p. 20
  • The experiment compares CPC-controlled CBO with a CBO baseline on a synthetic, two-dimensional continuous task with unknown binary feasibility constraints.The task uses the Townsend objective and a safe Gaussian policy centered at (0, -1).Source: B. Comparison to Conservative… · p. 20
  • Across temperatures 2.0, 1.0, and 0.1, CPC-controlled CBO keeps constraint violations below alpha=0.4 and attains comparable or improved reward over CBO without CPC.The CBO baseline exceeds alpha=0.4 across settings.Source: B. Comparison to Conservative… · p. 20, Figure 8 · p. 21

C.1.1. NOTATION AND DEFINITIONS

  • The notation appendix treats a bag as an unordered multiset and interprets replace-one stability using conditional sampling from a fixed bag.The described condition recovers standard average replace-one stability.Source: §C.1.1 · p. 22

C.1.2. PROOF FOR MAIN GENERALIZED CONFORMAL RISK CONTROL RESULT

The proofs establish risk-control guarantees for arbitrary policy-control parameters, including a likelihood-ratio-bound special case, by controlling oracle risk and transferring the bound to empirical selection using stability and continuity arguments. They also address non-monotonic weighted risk and show that bounded losses and a safe endpoint alone do not ensure risk control.

C.1.2. PROOF FOR MAIN GENERALIZED CONFORMAL RISK CONTROL RESULT

  • Conditioning on the data bag lets the proof establish deterministic oracle control of bag-conditional risk, then use exchangeability over permutations to relate that risk to the target expectation.The proof first reduces the target to a bound conditional on each bag; the oracle threshold is fixed given the bag.Source: §C.1.2 · pp. 23–24

Step 3: Relating the empirically-selected hyperparameter to the oracle parameter

  • The proof transfers oracle control to the empirical parameter by relating their thresholds and bounding the resulting risk difference with Lipschitz continuity and replace-one stability.The empirical threshold varies with the loss in the risk sum, so the proof uses stability and smoothness rather than applying the oracle bound directly.Source: Step 3: Relating the… · pp. 24, 26, 27

C.1.3. COUNTEREXAMPLE TO GCRC RISK CONTROL ON ARBITRARY BOUNDED LOSSES

  • For arbitrary bounded losses, boundedness and existence of a safe maximum hyperparameter are insufficient: the counterexample’s bag-conditional empirical risk equals 1.The example has bounded losses and loss zero at the maximum hyperparameter for every loss function.Source: §C.1.3 · p. 27

C.1.4. CONSERVATIVE ADJUSTMENT TO GCRC FOR LEVEL α VALIDITY IF K IS KNOWN

  • When the Lipschitz constant is known, a conservative significance-level adjustment yields level α validity.The supplied passages state the guarantee for the procedure run at the adjusted level, without showing the adjustment formula.Source: §C.1.4 · p. 28

C.2.1. NOTATION AND DEFINITIONS

  • The policy-control notation defines losses as functions of a control parameter and uses order-free data bags, with relative smoothness and replace-one stability assumptions.The policy family is K-log-Lipschitz in log β, and the control parameter is assumed to satisfy ε(K)-relative replace-one stability.Source: §C.2.1 · pp. 29–30
  • Under bounded losses, mutual absolute continuity, and replace-one stability, Theorem C.10 bounds risk by α plus a term that shrinks with greater stability and smoother policy changes.For arbitrary β, the theorem also assumes the selected parameter exists and the policy family is K-log-Lipschitz in log β.Source: §C.2.2 · pp. 30–31
  • Theorem C.11 recovers the likelihood-ratio-bound result as a special case, requiring replace-one stability but no smoothness assumption.Its assumptions include bounded losses, mutual absolute continuity, and existence of the selected parameter.Source: §C.2.2 · pp. 30–31

Step 3: Relating the empirically-selected hyperparameter to the oracle parameter

  • The proof seeks to show that the oracle parameter β* is at least the empirically selected parameter, although the oracle cannot be implemented because it requires the test-point loss.Conditioning on the observed data bag reduces this comparison to checking the empirical parameter for every possible omitted loss.Source: Step 3: Relating the… · pp. 32–33

Step 4: Analyzing the bag-conditional risk of the empirical procedure

  • Because changing unnormalized weights can lower re-normalized weighted risk, conservative weight bounds need not be conservative risk bounds; the proof therefore invokes smoothness and stability.This differs from bounding loss perturbations, where conservative loss estimates yield conservative risk estimates.Source: Step 4: Analyzing the bag-conditional… · p. 34
  • Marginalization and ϵ(K)-relative replace-one stability yield the claimed result; Theorem C.11 follows at K=1 using Lemma C.12 for the likelihood-ratio bound.Source: Step 4: Analyzing the bag-conditional… · p. 38
  • For considered values of β with ψ(β)>0, normalized clipped densities are uniformly 1-Lipschitz in log β for every x, and the constant 1 is sharp.Source: Step 4: Analyzing the bag-conditional… · p. 38

D. Detailed Methodology

The methodology details likelihood-ratio-bound calibration and sampling for constrained policies, alongside medical-QA factuality setups using false-discovery loss and true-claim recall. It also motivates generalized CRC for non-monotonic FDR loss and describes an experimental setup adapted from prior work.

D.1. Conformal Calibration of β

  • Calibration uses weighted conformal risk control to choose β, estimating ψ(β) by importance-weighted Monte Carlo and weighting for calibration-to-deployment distribution shift.The shared context distribution lets the context marginal cancel in the likelihood ratio, so calibration uses conditional densities at the same context.Source: §D.1 · p. 40

D.1.1. LIKELIHOOD RATIOS AND GRID PREPARATION

  • Calibration combines prior-round feasibility-labeled data with fresh samples from πt, builds a β grid from observed likelihood ratios, and stops at the first weighted infeasibility rate above α.The grid is searched in increasing order, from the most conservative candidate to the most permissive.Source: §D.1.1 · p. 40

D.1.3. CONFORMAL MIXTURE WEIGHT APPROXIMATION

  • The mixture-weight approximation replaces the joint distribution of past policies with a mixture weighted by calibration sample counts, then uses worst-case loss B for the unknown test point.It selects the largest β for which estimated risk remains at most α for every smaller candidate.Source: §D.1.3 · pp. 42–43

D.2.1. REJECTION SAMPLING WITH SIMPLE PROPOSALS

  • For simple rejection sampling, πsafe is often more efficient at small β, πt at large β, and neither necessarily dominates at intermediate β.At intermediate β, the constrained target may draw comparable mass from both policies, motivating mixture proposals.Source: §D.2.1 · p. 44

D.2.2. REJECTION SAMPLING WITH MIXTURE PROPOSAL

  • A mixture rejection proposal can weight πsafe and πt according to their estimated overlap with the constrained target, or adapt weights online toward the component with higher acceptance.The overlap-based heuristic retains support from both components while favoring the closer match.Source: §D.2.2 · p. 45

D.2.3. INDEPENDENCE METROPOLIS-HASTINGS SAMPLING

  • Independence Metropolis-Hastings avoids requiring a finite global rejection-sampling bound and yields a chain with the target as its invariant distribution.It compares each proposal with the current state rather than a global worst case.Source: §D.2.3 · p. 45

E.1. Problem Setup and Notation

  • Medical-QA samples pair prompts and LLM responses with extracted atomic claims and factuality annotations, and use prompt-conditional confidence scores to assess claims.Higher confidence scores indicate greater estimated factuality.Source: §E.1 · p. 46

E.3. Loss Functions for Factuality Control

  • For factuality control, prior methods threshold claim-confidence scores, while this work controls the false-discovery fraction among included claims instead of the loss for including any false claim.Conditional calibration and multicalibration in prior work target guarantees across prompt subgroups.Source: §E.3 · p. 46, §E.2 · p. 46
  • Because FDR loss can be non-monotonic in the threshold, standard CRC does not apply directly, motivating generalized CRC.Both false claims and total claims decrease as the threshold rises, but not necessarily at the same rate.Source: §E.3 · p. 47
  • Monotonizing the loss yields a non-increasing upper bound on FDR, but can be overly conservative.True-claim recall measures the fraction of true subclaims retained; higher recall preserves more useful information at a fixed FDR target.Source: §E.3 · p. 47

E.4. Experimental Setup

E.4.1. DATASET

The medical QA experiments evaluate factuality control using subclaims scored in three ways. Across the tested scoring methods, gCRC controls false discovery rate at the target level while retaining more true claims than baselines.

E.4.1. DATASET

  • MedLFQA aggregates four established medical QA benchmarks; the described examples include K-QA, LiveQA, and MedicationQA.Prompts have LLM- or human-generated reference responses used as ground truth for factual accuracy.Source: §E.4.1 · p. 47

E.4.2. DATA GENERATION PIPELINE

  • The data pipeline generates medical responses, extracts atomic subclaims, and assigns each subclaim a binary factuality label against a reference response.Self-evaluation scores are also used, with the LLM assessing its confidence in each claim.Source: §E.4.2 · p. 47

E.4.3. METHODS COMPARED

  • The comparison includes gCRC, CRC on monotonized losses, and Learn Then Test, which uses Hoeffding-Bentkus p-values for family-wise error control.LTT does not add a conservative test-point loss term.Source: §E.4.3 · p. 48

E.4.4. EXPERIMENTAL PROTOCOL

  • Experiments use a 70%/30% calibration/test split, 20 target risk levels, and 200 candidate thresholds, averaging results over 25 splits.Uniform noise of about 10^-8 is added to scores to break ties.Source: §E.4.4 · p. 48

E.5. Hyperparameters

E.6. Additional Medical QA Results

  • Across three subclaim scoring methods, gCRC tightly controls FDR at the target level and achieves superior recall to baselines.Results are averaged over 25 random splits.Source: Figure 10 · p. 49

F. Constrained Tabular Active Learning Implementation Details and Additional Results

The appendix details constrained active-learning experiments on three regression datasets and supplementary constrained sequence-optimization experiments using Ehrlich functions. The tabular setup creates synthetic feasibility constraints and applies CPC to an uncertainty-seeking acquisition policy.

F.1. Datasets

  • The tabular experiments evaluate regression on Robot Arm Kinematics, Airfoil Self-Noise, and Medical Expenditure Panel Survey datasets.The datasets range from 1,503 to 33,005 samples and from 5 to 107 features.Source: §F.1 · p. 50

F.2. Feasibility Constraint Construction

  • Synthetic feasibility is based on the first principal component: atypical low-PC1 records are treated as riskier, with feasibility probabilities tied to target risk α.The initial data is tilted toward high-PC1 records, while the remaining pool includes atypical records that uncertainty-based sampling may favor.Source: §F.2 · p. 50

F.3. Active Learning Setup

  • The active-learning setup initializes from high-PC1 data, then applies CPC to clip the likelihood ratio of an uncertainty-seeking acquisition policy relative to the source distribution.Each sampled record supplies a noisy label and feasibility indicator; observations are split between training and calibration before refitting the Gaussian Process.Source: §F.3 · p. 51

F.4. Hyperparameters

G. Constrained Black-Box Sequence Optimization Implementation Details and Results

G.1. Ehrlich Test Functions

  • Ehrlich functions score sequences by satisfaction of gapped motifs, assign infeasible sequences a score of −∞, and model expression constraints through a discrete Markov feasibility set.The experiments use Ehr(32, 32)-4-4-4: length-32 sequences, vocabulary size 32, four motifs of length four, and precision four.Source: §G.1 · p. 52

G.2. Training the Safe Policy

The sequence-optimization experiments train a safe policy from genetic-algorithm data, then improve it using DPO. Supplementary results report that moderate risk control may stabilize optimization and improve overall performance by wasting fewer samples on infeasibility.

G.2. Training the Safe Policy

  • The safe policy is trained by supervised fine-tuning on improving sequence pairs drawn from genetic-algorithm history, with diverse seeds selected from top-scoring feasible sequences.The initial genetic algorithm runs for 10 rounds with 10,000 particles per iteration, producing approximately 100,000 scored sequences.Source: §G.2 · p. 52, Table 4 · p. 52

G.3. Improving the Safe Policy

  • Policy improvement uses DPO preference triplets ranked by oracle scores, initializes from the safe-policy weights, and then applies CPC to the improved policy.The reported setup uses 10 seeds per DPO-post-trained policy; DPO was chosen as a challenging test because it struggles with poor feasibility rates relative to other fine-tuning approaches.Source: §G.3 · pp. 52–53, Table 5 · p. 53
  • Moderate CPC risk control may stabilize the Ehrlich optimization algorithm and improve overall performance by reducing samples wasted on infeasible solutions.Figure 11 tracks average feasibility, average objective value of DPO-policy samples, and best solution found over rounds; α directly adjusts constraint-violation risk.Source: Figure 11 · p. 53

Related papers

Loading 2603.02196v4…