PaperlayerView exact revision on arXiv

Source-linked AI summary

Certified Safety Curation: Distribution-Free Guarantees for Safe Offline Reinforcement Learning

Adam Haroon, Cody Fleming

arXiv:2609.12014v2cs.LG

TL;DR

When per-transition costs are unavailable, the paper asks what safe offline RL can still deliver from clip comparisons and occasional budget labels. It learns a state-only value to select and certify whole-trajectory training data before cloning; policies are safe on twelve of fifteen tasks, and certified selections also make a full-label baseline safe after retraining.

  • Problem

    Safe offline RL methods require a cost function on every dataset transition, but the paper asks what can be achieved from clip comparisons and occasional episode-budget judgments instead.

    Source: §1 · p. 1

  • Method

    A state-only value learned from segment comparisons scores whole trajectories, and Learn-then-Test calibrates a threshold to certify the selected training set before behavior cloning.

    Source: §1 · p. 1

  • Results

    The calibrated filter is safe on twelve of fifteen tasks, matching the ground-truth-safe-subset clone, and retraining the strongest full-label baseline on certified selections makes it safe on the four certifying tasks.

    Source: §1 · p. 2, §6.2 · pp. 6–7

  • Takeaways & Limitations

    The guarantee applies to training-set composition rather than policy cost, and the certified selection can be handed to another offline learner.

    Source: §8 · p. 9

    The certificate bounds training-set composition, not policy cost, and all preferences are synthesized from ground-truth episodic costs rather than collected from human labelers.

    Source: §7 · p. 9

Abstract

from arXiv · show

Safe offline reinforcement learning assumes a cost function on every transition. We ask what remains possible when safety can be judged only by comparing short clips and occasionally asking whether an episode exceeded its budget. Certified safety curation answers with a filter-then-clone pipeline: a state-only value trained from segment comparisons scores whole trajectories, Learn-then-Test calibration certifies a selection threshold under a distribution-free $(\alpha, \delta)$ bound on the unsafe fraction of the selection, and behavior cloning follows. What is certified is the training set, not the policy, whose cost we report rather than bound. Where no threshold attains the target, the procedure refuses. We are not aware of prior work certifying the composition of a training set for offline RL or imitation. Oracle controls justify the design: reweighting individual transitions fails even with an exact value, so the value selects whole trajectories. The policies satisfy the cost budget on twelve of fifteen DSRL tasks, matching a clone of the ground-truth safe subset, which needs a label on every trajectory. Retrained on the certified selection, the strongest full-label method becomes safe where no setting of its own cost target rescues it. Refusal is predictable: the certificate's probability has a closed form in the purity of the pool's top quantile, which the calibration sample estimates and through which the scorer enters.

1 INTRODUCTION

The paper asks whether clip comparisons and occasional episode-budget judgments can support safe offline RL when per-transition costs are unavailable. It proposes certifying a selected training set before cloning, while reporting policy cost separately.

Core question and approach

  • Existing safe offline RL methods require costs for every dataset transition, whereas practitioners may instead have pairwise clip-safety judgments and occasional episode-budget judgments.Recent preference-based safe offline RL work is described as lacking a statistical guarantee.Source: §1 · p. 1
  • The filter-then-clone pipeline uses a comparison-trained, state-only value to score whole trajectories and calibrates a selection threshold against 200 budget-exceedance labels.Learn-then-Test provides an (α, δ) bound on the selected set’s unsafe-trajectory fraction; if no threshold qualifies, the procedure refuses certification.Source: §1 · p. 1, Figure 1 · p. 2
  • Twelve of fifteen tasks are safe under the calibrated filter, while a ground-truth-safe-subset clone is also safe on twelve and requires a budget label on every trajectory.The paper’s contribution account also reports that the strongest full-label baseline becomes safe after retraining on certified selections.Source: §1 · p. 2

2 RELATED WORK

Prior work covers reward-oriented data filtering, safe offline RL with full costs or emerging weak supervision, and distribution-free risk control. The paper distinguishes its contribution by certifying training-set composition using preference-learned trajectory values.

Filtered and weighted behavior cloning

  • Reward-oriented filtering and weighting ranks trajectories or transitions for task success, but the cited approaches do not target safety constraints and use hand-tuned thresholds.Examples include percentile behavior cloning [Emmons et al., 2021] and advantage-based transition weighting [Peng et al., 2019].Source: §2 · p. 2

Safe offline RL

  • Safe offline RL includes full-cost methods and newer preference or weak-label methods, but none of the cited methods selects training data with a preference-learned value and a statistical guarantee.Examples of the latter include PReSa, OSIL [Burnwal et al., 2026], and PREFINE.Source: §2 · p. 2

Distribution-free risk control

  • Existing risk-control applications in RL and control attach guarantees to policies, while conformal selection controls expected false discoveries rather than the paper’s (α, δ) high-probability bound.The paper uses Learn-then-Test with an exact finite-population p-value and a closed form for certification probability.Source: §2 · pp. 2–3

3 PRELIMINARIES

The paper studies offline constrained MDPs with a static trajectory dataset and two weaker supervision signals in place of per-transition costs. It evaluates policies by reward and episodic cost relative to each task’s budget.

Problem setup

  • A trajectory is unsafe when its episodic cost, the sum of its per-step costs, exceeds the budget; policy safety means expected cost is within budget.The constrained MDP objective maximizes return subject to a cost-budget constraint.Source: §3 · p. 3
  • The method uses pairwise labels stating which of two length-H segments is safer and binary episode labels indicating budget exceedance for calibration.Calibration trajectories are sampled uniformly without replacement from the dataset, making the finite-population test exact.Source: §3 · p. 3

4 METHOD: CERTIFIED SAFETY CURATION

Certified safety curation learns a state-only value from segment comparisons, scores whole trajectories, and uses calibrated threshold selection before behavior cloning. Reward-aware curation is gated on certification.

4 METHOD: CERTIFIED SAFETY CURATION

  • The method has four stages: value learning, trajectory scoring, calibrated selection, and cloning.The procedure is described as a filter-then-clone pipeline.Source: §4 · p. 3

4.1 SAFETY VALUE FROM SEGMENT PREFERENCES

  • The safety value is trained with a Bradley-Terry model on segment sums, but preference supervision leaves per-state values underdetermined; ensemble disagreement diagnoses where the value is not identified.The ensemble is retained for that diagnostic role, while single members match its filter precision to within 0.006.Source: §4.1 · p. 3

4.2 TRAJECTORY SCORING AND SELECTION

  • Averaging state values over a trajectory yields its score, and thresholding selects whole trajectories for behavior cloning rather than fragmenting multi-step avoidance behavior through transition reweighting.Trajectory aggregation is used because it cancels per-state noise left unconstrained by the preference loss.Source: §4.2 · p. 3

4.3 CALIBRATED THRESHOLD WITH A COMPOSITION GUARANTEE

  • Learn-then-Test examines a decreasing grid of thresholds using an exact hypergeometric test and fixed-sequence stopping, returning the last rejected threshold or no certificate.The guarantee is transductive to the finite pool and unconditional over calibration draws, not a bound on violation conditional on receiving a certificate.Source: §4.3 · pp. 3–4

4.4 CERTIFICATION-GATED REWARD-AWARE CURATION

  • Return-based curation is enabled only for certified selections, because return and cost correlate on contaminated selections; the method offers both top-half filtering and clipped return-weighted cloning.The weighting intensity is controlled by κ, and safety beyond the certified composition is not guaranteed by this passage.Source: §4.4 · p. 4

5 WHY TRAJECTORY SCALE: AN ORACLE-CONTROLLED ANALYSIS

Oracle controls show that transition-level reweighting can fail even with exact values, while preference-trained values are reliably identified at trajectory scale. Selecting whole trajectories preserves multi-step behaviors and can replace full labeling with a preference-learned score.

The obstacle: per-transition reweighting

  • Exact-value transition reweighting fails to improve over the learned value in any of nine tasks, and reaches costs of 235 and 169 against budget 20 on HalfCheetah and Ant.The per-transition route is not an absolute failure: five of eleven configurations meet budget on PointGoal1, but none of nine do on CarGoal2 or PointGoal2.Source: §5 · p. 5
  • Trajectory-scale selection keeps whole multi-step avoidance behaviors intact, whereas per-transition weighting fragments them; the analysis proves a case where that failure is unavoidable.The paper defines trajectory-scale use as aggregating value over a whole trajectory and selecting on that aggregate.Source: §5 · pp. 4–5

Identification at per-transition scale

  • Preference-trained values rank trajectories reliably, but their top-one-percent per-transition overlap varies from 0.39 to 0.92 across tasks and seeds.Neither held-out accuracy nor per-transition reliability predicts policy performance; the value is identified at trajectory scale, not reliably at per-transition scale.Source: §5 · p. 5

Trajectory-scale selection

  • Cloning the ground-truth safe subset meets budget on eight of nine tasks, including PointGoal1, where no per-transition operator succeeds.The proposed method replaces that full labeling with a preference-learned score and adds certification; value diagnostics also recover cost-relevant state features and navigation hazard fields.Source: §5 · p. 5

6 EXPERIMENTS

Across fifteen DSRL tasks, the experiments compare preference-based trajectory filters and clones with full-label filters, policy-optimization baselines, and matched-size controls. They assess budget satisfaction, certification behavior, transfer, composition with other learners, and ablations.

6.1 SETUP

  • The fifteen-task evaluation uses five velocity-constrained locomotion and ten SafetyGymnasium navigation tasks, with shared architectures, evaluation protocol, and preference data.Headline configurations report means and 95% bootstrap confidence intervals over five seeds.Source: §6.1 · p. 5

6.2 MAIN RESULTS

  • The calibrated filter is safe on twelve of fifteen tasks, including its uncertified fallback, but its certificate fires on four tasks at α = 0.25.The V-filter at the calibration-estimated fraction is also safe on twelve of fifteen, matching BC-Safe, which requires a budget label on every trajectory; their failures differ.Source: §6.2 · p. 6
  • The certification-gated reward-aware variant stays within budget on all four certified tasks, while its ungated version raises mean cost to 82 on contaminated Hopper selections.The unchanged pipeline also meets budget on all five BulletSafetyGym transfer tasks, with the guarantee holding in every validation cell.Source: §6.2 · pp. 6–7
  • Trajectory-scale filter-then-clone arms keep eleven to thirteen tasks within budget, versus two to five for optimization-based methods; segment-scale filtering with full labels keeps two of fifteen.The matched-size controls without useful selection signal do not match the trajectory-scale results.Source: §6.2 · p. 7
  • Retrained CDT is safe in 36 of 36 runs across the four tasks certified at α = 0.25, although no cost target rescues full-data CDT on navigation.CPL is within budget in all 27 runs on three of the four tasks, but certified curation pushes it beyond budget on Walker2d.Source: §6.2 · p. 7
  • Certified return-weighted cloning is safe in 39 of 40 within-specification cells; on HalfCheetah it raises reward from 2104 to 2271 at cost 0.03.The exception is CarRun’s hard top-half rule; outside certified selections, the operator violates once the clip is loosened to κ = 3.Source: §6.2 · p. 7

6.3 GUARANTEE VALIDATION

  • The unconditional false-certification rate stays below δ = 0.1 in every validation cell, with a worst value of 0.078; certification frequency tracks the pool’s purity margin.Certification can remain rare or never occur when no threshold achieves the target, and the certificate concerns training-set composition rather than policy cost; cloning can still violate budget.Source: §6.3 · pp. 7–8

6.4 ABLATIONS

  • Matched-size return selection is unsafe on all fifteen tasks and random selection on fourteen; certification becomes more frequent with larger calibration samples and looser α.Preferences yield one more safe task than labels-only filtering at matched budget—twelve versus eleven—but no additional certification; five further controls find that numeric costs buy nothing and action-conditioned Q forfeits policy-agnostic scoring.Source: §6.4 · p. 9

7 LIMITATIONS

The guarantee applies to training-set composition, not the resulting policy’s cost. Its scope also depends on labeled calibration data and synthesized preferences, while tail-robust scoring remains open.

Limitations

  • The certificate covers training-set composition, not policy cost; a starved selection can underfit into an unsafe policy despite the fallback’s minimum-size safeguard.The guarantee uses 200 labeled trajectories; the preference-only variant removes those labels but also removes the certificate.Source: §7 · p. 9
  • Preferences are synthesized from ground-truth episodic costs, so the weak-oracle regime is modeled under label noise and a Boltzmann labeler rather than tested with human labelers.The extreme trajectory-score tail is unreliable; capped thresholds work around it, while tail-robust scoring remains open.Source: §7 · p. 9
  • Composability is evaluated on four tasks at α = 0.25 and eight at α = 0.40; the action-conditioned ablation covers five tasks.A labels-only classifier comes within one safe task of the comparison route, whose advantage is mainly the held-out audit rather than coverage.Source: §7 · p. 9

8 CONCLUSION

Certified safety curation selects whole trajectories with a state-only value learned from segment comparisons, calibrates a threshold, and clones the certified selection. The certificate covers training-set composition rather than policy behavior.

8 CONCLUSION

  • The strongest full-label baseline becomes safe when retrained on the certified selection, where it otherwise violates the budget.The certificate is available before a policy is trained or run.Source: §8 · p. 9
  • Certified safety curation uses a state-only value trained from segment comparisons to select whole trajectories, calibrates the threshold with Learn-then-Test, and behavior-clones the certified selection.The guarantee is distribution-free and concerns the composition of the selected training set.Source: §8 · p. 9
  • Generative AI assisted with experiment-orchestration and analysis scripts, table and figure regeneration, and prose editing; authors reviewed code and cross-checked outputs against archived records.The paper states that AI did not originate the research question, method, theoretical results, or proofs, and did not produce or alter experimental results outside the logged pipeline.Source: AI USE STATEMENT · p. 10
  • The certificate bounds training-set composition, not the resulting policy’s behavior; the paper reports certification and policy outcomes separately.The study uses public simulated benchmarks, and preference labels are synthesized from benchmark costs rather than elicited from people.Source: ETHICS STATEMENT · p. 10
  • Experiments use public DSRL datasets, archived records regenerate tables, and accompanying scripts regenerate guarantee figures from archived calibration statistics.The CarRun composability echo is an exception: its original certified selection was lost and re-derived.Source: REPRODUCIBILITY STATEMENT · p. 10

A ALGORITHM

  • The algorithm trains an ensemble value on preference data, labels calibration trajectories, and applies fixed-sequence Learn-then-Test across score quantiles, returning the last certified threshold or an explicit no-certificate report.After certification, it may curate for reward before behavior cloning.Source: A ALGORITHM · p. 12

B PROOFS

The proofs establish finite-pool validity for the selection-composition certificate and give an exact expression for its certification probability. A corridor construction explains why whole-trajectory selection can preserve safety when per-transition reweighting cannot.

B PROOFS

  • Conditioning on the calibration sample’s overlap with a selection gives a hypergeometric unsafe count, whose super-uniform p-values make false certification probability at most δ.The fixed stopping rule cannot reach a true null without first rejecting it, so the bound holds without monotonicity of unsafe fractions across thresholds.Source: B PROOFS · p. 13
  • Although the scorer is trained using labels from the same pool, validity holds because the score and selections are fixed before the calibration draw.The guarantee concerns selection composition, not a property of the scorer; calibration trajectories labeled unsafe could be excluded to improve composition, but the reported results do not do so.Source: B PROOFS · p. 13
  • Resampling a certified selection with weights in [e−κ, eκ] changes its unsafe-mass bound to e2κα/(e2κα+1−α), which is informative only for small κ.At α = 0.25, this bound already exceeds one-half by κ = 1; direct certification of weighted risk exceeds the reported label budget.Source: B PROOFS · pp. 13–14
  • Refusal follows when the first threshold is too impure: certification is unlikely if its unsafe fraction exceeds α, while its probability approaches one as the calibration sample grows when it is below α.The purity margin predicts attainability and label cost; refusal is therefore a correctness property, not merely a conservative heuristic.Source: B PROOFS · p. 14
  • In a deterministic corridor, state-only per-transition weights lose all-safe probability with horizon, while selecting zero-cost trajectories yields exact safety with probability at least 1−(1−q)^N.The construction separates the operator classes: bounded weight clips fail beyond a horizon threshold, whereas selection succeeds whenever the safe expert subset appears in the dataset.Source: B PROOFS · pp. 14–15
  • The fixed-sequence walk certifies if and only if it rejects the first, most selective threshold, reducing certification probability to one hypergeometric event.Equation 3 sums over the calibration sample’s overlap with the selection and the unsafe count within that overlap; its expression also yields a calibration-label budget for a target rate.Source: B PROOFS · pp. 15–16
  • Across 198 constructed pools, certification stayed at most 0.08 for non-positive purity margins and rose to one by a margin of about 0.2.The sweep reused deployed scorers and changed only pool membership; Equation 3 tracked the rates with mean absolute error 0.010 and correlation 0.999.Source: B PROOFS · p. 17, Figure 3 · p. 17

C FULL EXTRACTION SWEEP

The extraction sweeps show that per-transition value use remains unreliable even with oracle values, while trajectory-level aggregation and selection better preserve safety-relevant structure. Weight and aggregation variants qualify that result: stronger reweighting can trade reward for cost, and alternative trajectory aggregators have task-specific effects.

C FULL EXTRACTION SWEEP

  • In the full oracle-value ablation, no oracle improves on the learned value in any cell; the closest is CarGoal1, where discounted cost-to-go costs 19.43 versus the learned value’s 14.00, both within budget.Source: C FULL EXTRACTION SWEEP · pp. 17–18
  • On the three failing navigation tasks, no weight transform escapes the cost plateau, and the action-conditioned oracle remains over budget at 39.5, 31.5, and 49.6 against 25.The oracle raises reward above every weight transform, but varying extraction temperature reaches budget only on PointGoal1 among those tasks.Source: C FULL EXTRACTION SWEEP · p. 18, Table 5 · p. 19
  • Dense per-step supervision also yields lower cross-seed advantage-rank agreement than preference training on identical transitions across all nine tasks, so per-transition disagreement is not specific to preference supervision.Task-mean rank correlations are 0.43 to 0.82 for dense supervision versus 0.79 to 0.99 for preference ensembles.Source: C FULL EXTRACTION SWEEP · p. 18
  • Trajectory-level held-out accuracy ranges from 0.798 to 1.000, while per-transition top-one-percent overlap ranges from 0.39 to 0.92, so the tail emphasized by exponential weighting is seed-specific.Held-out accuracy measures agreement on pairs drawn from the bottom and top cost quartiles; transition rows compare seed pairs on 5,000 matched transitions.Source: Table 3 · p. 18, C FULL EXTRACTION SWEEP · p. 19

D EXTRACTION TEMPERATURE, WEIGHT CLIP, AND SCORE AGGREGATION

  • On PointGoal1, β = 0.1 reaches cost 16.40 against budget 25, but reward falls to 10.80 and cost remains 16.40 against 8.80 for trajectory selection.CarGoal2 and PointGoal2 remain over budget at every swept setting; the PointGoal1 improvement requires a reward reduction from 17.53 to 10.80.Source: D EXTRACTION TEMPERATURE, WEIGHT CLIP,… · pp. 19–20, Table 6 · p. 20
  • The minimum aggregator is safe on four of nine tasks versus eight for the mean; the tenth percentile also reaches eight, rescuing Hopper at cost 19.7 versus the mean’s 33.0.The tenth percentile loses Swimmer, making its gain a preregistered improvement on one task rather than a sweep-wide gain.Source: D EXTRACTION TEMPERATURE, WEIGHT CLIP,… · p. 20, Table 7 · p. 20

E VALUE DIAGNOSTICS: ANTICIPATION, TRACES, AND THE SAFETY

The diagnostics show that the learned value is noisy state by state but carries a graded safety signal in aggregate. External-baseline comparisons place that behavior alongside methods with stronger supervision, which do not exceed the calibrated filter's task-level safety count.

E VALUE DIAGNOSTICS: ANTICIPATION, TRACES, AND THE SAFETY

  • The ensemble mean falls by two to four within-task standard deviations between distant states and imminent violations across all nine tasks, despite individual bins reversing by up to 0.9 standard deviations.In a single unsafe trajectory, per-state values swing across several units while their rolling mean remains stable through the episode’s cost-event bands.Source: E VALUE DIAGNOSTICS: ANTICIPATION,… · p. 20, Figure 4 · p. 21
  • On three PointGoal1 layouts, value-hazard distance correlations are 0.43, 0.44, and 0.20, and fall to 0.09, 0.27, and 0.15 when states are frozen at zero velocity.The value therefore aligns with proximity and motion jointly on visited states rather than position alone; in the second layout, the goal lies within the hazard field.Source: E VALUE DIAGNOSTICS: ANTICIPATION,… · pp. 20–21, Figure 5 · p. 21

F EXTERNAL BASELINES AND LABELING COST

  • No external baseline exceeds the calibrated filter’s twelve safe tasks: CDT reaches seven of fifteen, while BC-Safe reaches twelve using ground-truth labels.BC-Safe-Seg is safe on two of fifteen despite full labels, consistent with trajectory-scale selection preserving multi-step avoidance behaviors better than segment-scale selection.Source: F EXTERNAL BASELINES AND LABELING COST · p. 22, Table 9 · p. 22, Figure 6 · p. 23
  • CDT, CPQ, and COptiDICE keep five, four, and two of fifteen tasks within budget at budget-matched limits; CPQ’s safe cells mostly pair near-zero cost with collapsed reward.COptiDICE’s two safe cells give up 23 to 36 percent of BC-All’s reward.Source: F EXTERNAL BASELINES AND LABELING COST · p. 22, Table 8 · p. 22

G COMPOSABILITY AND CONTAMINATION TOLERANCE

Composability depends on both the certified selection and the learner consuming it: tighter selections support stronger safety outcomes, while wider contamination bands expose learner-specific limits. Across archived selection-clone pairs, contamination tracks clone cost but does not explain it completely.

G COMPOSABILITY AND CONTAMINATION TOLERANCE

  • At α = 0.40, our clone is safe on 22 of 24 task-selections, while CDT is safe on 14 of 24 and violates on every navigation task on at least one selection.The clone’s two failures are Swimmer tail selections; PointGoal2 is non-monotone in contamination, so contamination is a strong tendency rather than a law.Source: G COMPOSABILITY AND CONTAMINATION… · p. 24, Table 12 · p. 25, Table 13 · p. 25
  • At α = 0.25, CDT stays within budget on all 36 runs across twelve distinct selections, while CPL is within budget on 28 of 36, with all eight failures on Walker2d.The three selections per task include three whose realized contamination exceeds the certified level.Source: Table 11 · p. 24, G COMPOSABILITY AND CONTAMINATION… · p. 24
  • Across 235 archived selection-clone pairs, task-demeaned contamination explains R2 = 0.41 of clone cost; adding score-margin statistics raises R2 to 0.49.The remaining variance is partly associated with learner effects characterized by the composability grids.Source: G COMPOSABILITY AND CONTAMINATION… · p. 25, Figure 7 · p. 26
  • Of 40 within-specification certified return-weighted cells, 39 are safe at every intensity; the sole exception is CarRun’s hard top-half rule.Among twelve out-of-specification cells, eleven are safe; the most contaminated HalfCheetah selection violates only at κ = 3, with cost 25.1 against budget 20.Source: G COMPOSABILITY AND CONTAMINATION… · pp. 25–26, Table 14 · p. 26

H CDT COST-TARGET SWEEP

CDT’s cost-target sweep separates tasks where conditioning delivers safety at high reward from tasks where the full dataset does not support the requested return-cost tradeoff. On Goal tasks, the calibrated filter’s α sweep instead spans a within-budget frontier.

H CDT COST-TARGET SWEEP

  • On the eight Goal and Button tasks, CDT violates at every cost target and its realized cost is nearly flat, including 40.9 to 37.8 on CarGoal1 against budget 25.Across the nine-task analysis sweep, Figure 8 shows the full CDT curve to the right of the budget on Goal tasks while the calibrated α sweep traverses a frontier left of it.Source: H CDT COST-TARGET SWEEP · p. 26, Table 16 · p. 27, Figure 8 · p. 28
  • Where the pool admits it, below-budget conditioning yields high reward: all five velocity tasks and both Circle tasks, plus all five BulletSafetyGym tasks at target 2, exceed every filter’s reward.The BulletSafetyGym tasks use budget 10, while the DSRL tasks use budgets of 20 or 25.Source: H CDT COST-TARGET SWEEP · p. 26, Table 17 · p. 27
  • The paper attributes flat realized costs to datasets lacking the return-cost frontier CDT fits, so lowering its requested cost cannot produce trajectories the data never demonstrated.The appendix sweep retrains full-data CDT at each task’s budget and evaluates checkpoints across targets down to one-fifth of budget.Source: H CDT COST-TARGET SWEEP · pp. 26–27

I TRANSFER TO BULLETSAFETYGYM

On five BulletSafetyGym tasks, the calibrated filter transfers without per-task tuning and meets the budget on all five, but DroneRun exposes a case where safety comes with severe reward collapse. Certification varies by task, and curation does not always rescue a full-label learner when the pool itself is benign.

I TRANSFER TO BULLETSAFETYGYM

  • The score tracks episodic cost strongly, with Spearman correlations from −0.81 to −0.94 on every task, while random matched-fraction selection is unsafe on four of five.Random selection is safe only on CarRun, whose pool is 46 percent safe.Source: I TRANSFER TO BULLETSAFETYGYM · p. 28
  • All five calibrated-filter clones meet budget 10, while cloning all data is unsafe on four of five tasks; DroneRun’s safe result comes with reward collapse.On DroneRun, reward falls from 383 under unfiltered cloning to 42 for the calibrated filter; the estimated-fraction V-filter also meets budget on all five tasks.Source: I TRANSFER TO BULLETSAFETYGYM · p. 29
  • CarRun certifies on five of five seeds and DroneRun on one, while BallRun, BallCircle, and CarCircle refuse on every seed.The 2000-draw validation reports a worst unconditional false-certification rate of 0.06.Source: I TRANSFER TO BULLETSAFETYGYM · p. 29
  • On CarRun, curation does not reliably rescue CDT because the whole-pool clone is already safe; the remaining violation is attributed to the learner’s operator, not the data.On the regenerated certified selection, CDT violates on one of three seeds, whereas return-weighted variants are safe on every seed.Source: I TRANSFER TO BULLETSAFETYGYM · p. 29

J PROPERTIES OF THE CERTIFICATE

The certificate’s validity is maintained across calibration draws, while its yield and practical informativeness depend on how often the score’s selected pool can meet the purity target. The paper also tests certificates on executed policies, finding that deployment-time evidence complements—but does not replace—the offline composition guarantee.

J PROPERTIES OF THE CERTIFICATE

  • The unconditional false-certification rate stays below δ across tested cells, with a worst cell of 0.078 at CarGoal1, seed 1, n = 400.Its Clopper-Pearson interval is 0.067 to 0.091; Figure 9 reports every individual task, seed, and calibration-size cell below δ.Source: J PROPERTIES OF THE CERTIFICATE · p. 29, Figure 9 · p. 30
  • Certification rate is a trust signal: conditional violation can approach one on tasks that certify in under five percent of draws, despite the unconditional guarantee.Among tasks with defined conditional rates, the reported range begins at 0.07 for PointGoal1.Source: J PROPERTIES OF THE CERTIFICATE · pp. 29–30
  • The returned certificate triple estimates the certification rate well when certification is common, but overstates it on rare-certification tasks and its conditional bound then often misses.Coverage occurs in all 24 cells certifying in at least a tenth of draws, but only 25 of 40 cells overall.Source: J PROPERTIES OF THE CERTIFICATE · pp. 30–31
  • At n = 200, the fixed-sequence test certifies more often than Bonferroni or Holm while preserving validity; certification yield is limited by per-test power.PointGoal1’s rate is 0.654 for fixed-sequence versus 0.295 for each alternative; the worst false-certification rate is 0.063 against δ = 0.1.Source: J PROPERTIES OF THE CERTIFICATE · pp. 31–32
  • Calibration budgets track the purity margin: across seven tasks that reach half-certification, log n50 = −1.61 log(margin) + 1.66, while non-positive-margin tasks certify at essentially zero.Swimmer’s positive margin of 0.014 does not reach half-certification even at n = 3200.Source: J PROPERTIES OF THE CERTIFICATE · p. 32, Table 21 · p. 32
  • The uncertified fallback is deliberately conservative: it helps on contaminated Hopper but removes useful data on already-clean Swimmer.The smaller selection lowers Hopper’s clone cost from 33 to 3, but costs Swimmer its budget.Source: J PROPERTIES OF THE CERTIFICATE · p. 32

K CERTIFICATES ON THE DEPLOYED POLICY

  • Deployment-time policy certificates use executed episodes and answer a different question from the offline composition certificate; expected-cost certification succeeds on ten of fifteen tasks.The 100-episode tail bound is not the paper’s mean-cost criterion; 2000 fresh episodes are used for the mean-cost certificate, and selection statistics explain only about a tenth of cost variance.Source: K CERTIFICATES ON THE DEPLOYED POLICY · p. 33
  • Comparing 2000-episode estimates with 100-episode estimates, all fifteen safe-or-unsafe verdicts remained unchanged, with twelve safe tasks under both budgets, replicating Table 1’s result at twenty times the evaluation budget.Separately, using 2000 fresh episodes per cell on the same reported policies without retraining, expected cost was certified within budget on ten of fifteen tasks on every seed at δ = 0.1.Source: K CERTIFICATES ON THE DEPLOYED POLICY · p. 33

L.1 CONTROLS ON THE SCORE AND THE SELECTION

The controls separate supervision quality from the choice of what the scorer ranks and what the certificate can establish. They support trajectory-level value selection, show that labels-only filtering is competitive but consumes certification labels, and document the settings and limits of these comparisons.

L.1 CONTROLS ON THE SCORE AND THE SELECTION

  • Return ranking buys reward but has higher cost than random selection on twelve of fifteen tasks; safety tracks whether the selected fraction stays within the pool’s safe mass.At a fixed top-25-percent selection, the filter is safe on ten of fifteen tasks; Hopper’s 25-percent selection costs 60 versus 20 at its ground-truth-matched fraction.Source: §L.1 · pp. 34–35, Table 25 · p. 36
  • The action-conditioned oracle route remains unsafe on its documented configuration, with costs 31.5, 39.5, and 49.6 against budget 25, despite access to full cost labels and a long-horizon value.On PointGoal2 it obtains reward 12.2 against 6.2 for the selection route, illustrating that the per-transition operator buys reward rather than safety in this control.Source: §L.1 · p. 35
  • Across segment lengths, the deployed filter is safe on 8 of 9 tasks at H = 30 and 50, versus 7 of 9 at H = 10; certification changes with the purity margin left attainable.Mean certification rates are 0.224, 0.204, and 0.102 at H = 30, 50, and 10, respectively; across 27 cells the margin-rate correlation is 0.994.Source: §L.1 · p. 36

L.2 WHAT THE SUPERVISION BUYS

  • Preference noise degrades score precision without breaking the certificate, because validity comes from the clean calibration sample rather than score quality.At up to 30 percent noise, the worst false-certification rate reaches δ only once; precision falls by 0.05 to 0.14 on navigation tasks and 0.38 on Swimmer.Source: §L.1 · p. 34, §L.2 · p. 37
  • Labels-only filtering is safe on 11, 12, 11, and 13 of fifteen tasks at budgets n = 50, 100, 200, and 400, but using those labels for training leaves no same-sample certificate.Matching the preference route’s deliverable requires a second labeled sample or splitting the label budget; preferences provide scoring while budget judgments remain held out.Source: §L.2 · pp. 37–38
  • At matched budget, numeric-cost supervision yields a clone safe on twelve tasks versus eleven for binary labels, while its selection is less pure on eleven tasks.The average purity difference is 0.024, and the one-task safety difference is within draw-to-draw variation.Source: §L.2 · p. 38, Table 26 · p. 38
  • Labels-only filtering works in these tasks because trajectory safety is carried by low-dimensional, strongly separable state features, not simply because each episode labels many states.Reducing sampled states per labeled trajectory a hundredfold leaves the safe-task count essentially unchanged.Source: §L.2 · pp. 38–39

N HYPERPARAMETERS

  • The implementation uses three value networks and 1000 preference pairs per task, then calibrates at α = 0.25 and δ = 0.1 with n = 200 labels.The threshold grid spans score quantiles 0.85 to 0.30; the uncertified fallback uses a one-sided Clopper-Pearson lower bound on pool safe mass.Source: N HYPERPARAMETERS · pp. 39–40
Loading 2609.12014v2…