Safety Hacking in Constrained Best-of-$N$ Inference-time Scaling

Akifumi Wachi; Takumi Tanabe; Youhei Akimoto

Source-linked AI summary

Safety Hacking in Constrained Best-of-$N$ Inference-time Scaling

Akifumi Wachi, Takumi Tanabe, Youhei Akimoto

arXiv:2608.22915v1Aug 24, 2026cs.LGcs.AIcs.CLcs.CR

TL;DR

The paper studies how learned safety filtering and reward maximization interact in inference-time scaling. It formalizes safety hacking, derives finite- and asymptotic guarantees for constrained Best-of-N, and analyzes coverage control with constrained pessimistic sampling. The results show that unsafe reward tails can make hacking nearly certain despite small average proxy errors, while coverage control limits amplification without removing contamination.

Problem
Inference-time pipelines use imperfect learned safety and reward proxies, but the interaction between safety-filter contamination and downstream reward maximization is not fully characterized.
Method
The paper formalizes safety hacking, derives finite-N bounds from joint safe-versus-unsafe reward tails, and studies coverage control through constrained pessimistic sampling.
Results
A heavier unsafe reward tail can make safety hacking asymptotically certain even when false-positive mass and average proxy errors vanish.
Takeaways & Limitations
Coverage control limits reward-driven amplification but cannot repair unsafe outputs already admitted by the safety proxy.
Takeaways & Limitations
The theory analyzes fixed, non-adaptive pipelines with independently sampled candidates and does not cover adaptive search, self-refinement, tree search, or agentic planning.

Abstract

from arXiv · show

Inference-time pipelines often sample multiple outputs, filter them with a learned safety model, and return the proxy-feasible output with the highest learned reward. We show that this composition creates a two-stage failure: an imperfect safety proxy first contaminates the feasible set with unsafe outputs, and reward maximization can then amplify this residual contamination. We define \emph{safety hacking} as selecting an output that passes the learned constraint but violates the true safety criterion. For constrained Best-of-$N$ sampling, we derive finite-$N$ bounds governed by the joint upper reward tails of safe and unsafe outputs within the proxy-feasible set. If unsafe-but-feasible outputs have the heavier tail, safety hacking becomes asymptotically certain as $N$ grows, even when false-positive mass and average safety- and reward-proxy errors are arbitrarily small. We also show that policies within a bounded $χ^2$ divergence from the proxy-feasible reference distribution admit an $N$-independent safety-hacking bound, and instantiate this general coverage-control principle with constrained pessimistic sampling. Coverage control limits amplification but cannot repair a contaminated feasible set: admitted unsafe outputs may still be favored, and regularized selection is not necessarily safer than constrained Best-of-$N$ for every reward proxy. Toy and language-model experiments characterize both contamination and its reward-tail amplification, which exposes an inherent difficulty in inference-time scaling with learned safety models.

1. Introduction

Inference-time safety pipelines can fail when an imperfect safety filter admits unsafe outputs and reward maximization preferentially selects them. The paper formalizes this interaction and develops bounds, mitigation principles, and experiments for characterizing it.

Safety hacking occurs when a response passes the learned safety constraint but violates the true safety criterion.
CBoN can amplify small safety-filter contamination because admitted unsafe outputs remain eligible for downstream reward optimization.
The paper derives finite-N bounds based on the joint upper reward tails of safe and unsafe proxy-feasible outputs.
When unsafe outputs have the heavier learned-reward tail, safety hacking can become asymptotically certain even with vanishing false-positive mass and average proxy errors.
Policies with bounded χ2 divergence from the proxy-feasible reference distribution admit N-independent bounds, instantiated through constrained pessimistic sampling.
Controlled and language-model experiments measure contamination and reward-tail amplification, including reward-model ablations and an alternative safety filter.

2. Related Work

Related work covers reward overoptimization, safety-constrained inference, pessimistic inference, learned safety filters, and jailbreak evaluation. This paper distinguishes its focus on hard safety-filter misspecification and amplification within the contaminated feasible set.

Reward overoptimization: Reward overoptimization studies how maximizing an imperfect reward proxy can produce unintended behavior or reduce true reward.
Safety-constrained inference-time alignment: Prior safety-constrained inference work optimizes learned reward and cost signals, whereas this paper studies a misspecified hard safety filter.
Inference-time pessimism and coverage control: Pessimistic and regularized Best-of-N methods address coverage or scaling, while this paper conditions the reference distribution on proxy feasibility and analyzes residual contamination.
Learned safety filters and evaluator errors: Learned filters and guardrails operationalize safety constraints but remain imperfect proxies for the true criterion.
Jailbreaks and red teaming: Jailbreak benchmarks and BoN jailbreaking study harmful behavior and repeated attacks, while this paper fixes the prompt and analyzes filtering plus reranking of sampled outputs.

3. Problem Statement

The problem statement models inference with unknown true reward and safety functions, a reference policy, and learned reward and safety proxies. It separates proxy-feasible safe and unsafe outputs and defines safety hacking operationally.

The formulation uses a fixed prompt, an output space, an unknown reward function, and an unknown binary safety function.
The ideal constrained inference problem is approximated because the agent cannot directly evaluate the true reward or safety functions.
The agent instead samples from a reference policy and uses learned reward and safety proxies to select outputs.
The proxy-feasible set decomposes into truly safe accepted outputs and unsafe false-positive outputs.
Safety hacking is the event that an inference procedure outputs an unsafe response that satisfies the learned safety constraint; abstention is excluded.
The analysis defines joint learned-reward tails that combine reference-policy mass with upper-tail behavior within safe or unsafe proxy-feasible classes.
The trajectory formulation requires N i.i.d. samples and excludes beam search, Monte Carlo tree search, and other non-i.i.d. or stepwise procedures.

4. Safety-Filter Contamination and Reward-Tail Amplification

CBoN safety hacking has two stages: unsafe outputs enter the proxy-feasible set, then reward maximization searches for them among extreme learned-reward values. Finite-N bounds and asymptotic results quantify this amplification.

CBoN samples i.i.d. candidates, retains proxy-feasible ones, and returns the feasible candidate with the highest learned reward; it abstains when none is feasible.
The failure separates contamination, which determines unsafe feasible mass, from amplification, which results from reward-based competition within the feasible set.
Finite-N bounds characterize safety-hacking probability through upper-tail events for safe and unsafe proxy-feasible outputs.
The lower bound corresponds to at least one unsafe candidate exceeding a threshold while no safe candidate exceeds it.
At unsafe tail mass 0.01 above the safe maximum, the lower bound is approximately 0.63 at N = 100 and 0.99 at N = 500.
If safe and unsafe tails separate so that safe exceedances vanish while unsafe exceedances diverge, the safety-hacking probability converges to one.
For every ε, proxy errors can scale as √ε while N ≥ ε^-1 log(1/δ) still yields safety hacking with probability at least 1 − δ.
Heavier Gaussian upper-tail overestimation for unsafe outputs can also produce tail separation when true-reward ranges are bounded.

5. Coverage Control Limits Amplification, Not Contamination

Coverage control bounds how strongly inference can concentrate on the proxy-feasible reference distribution, making safety-hacking risk independent of N. It limits amplification of admitted unsafe outputs but cannot remove contamination.

Bounded deviation from the proxy-feasible reference distribution limits downstream amplification as N grows, without removing admitted unsafe mass.
The proxy-feasible reference distribution is obtained by conditioning the reference policy on passing the learned safety filter.
The coverage theorem applies to any inference procedure with bounded proxy-feasible coverage, regardless of how its final policy is constructed.
5.1. A Regularized Instantiation: cPes is a finite-sample reweighting algorithm that replaces population expectations with empirical analogues over proxy-feasible candidates.
5.1. A Regularized Instantiation: cPes provides an N-independent safety-hacking bound, including unconditional sample bounds when an abstaining fallback is used.

6. Experiments

Toy and language-model experiments show that reward maximization amplifies unsafe outputs admitted by the safety filter, while the magnitude of amplification depends on the reward proxy and is reduced by cPes.

6.1. Toy Problem: 0.023 to 0.9996: CBoN’s toy conditional safety-hacking rate rises as N increases from 1 to 8192, while mean true reward falls from about 0.786 to 0.200.
6.2. LLM Experiments: The LLM decomposition separates unsafe-only exposure from competitive unsafe wins, with the latter primarily driving increased safety hacking as N grows.
6.2. LLM Experiments: CBoN improves proxy reward but worsens safety and safety-aware reward with larger N, whereas cPes limits these effects.
6.2. LLM Experiments: 9.1% to 13.4%: With Beaver, CBoN safety hacking increases from N = 1 to 256, but with Skywork it decreases to 6.8% at N = 256.
6.2. LLM Experiments: At N = 256, Skywork CBoN has lower safety hacking than Skywork cPes, 6.8% versus 8.7%, so cPes lacks pointwise dominance.

7. Discussion and Limitations

Coverage control limits reward-driven concentration on residual unsafe mass but cannot remove contamination or guarantee dominance over CBoN. Robust inference therefore requires separate interventions on filtering, concentration, and reward-tail behavior, while the theory remains limited to fixed non-adaptive pipelines.

Discussion: Coverage control limits concentration on unsafe false positives but cannot identify or eliminate them.The proxy-feasible reference may already assign mass to unsafe outputs, and cPes can still favor those with high learned rewards.
Discussion: An N-independent safety-hacking bound implies neither absolute safety nor pointwise dominance over CBoN.The Skywork ablation illustrates this boundary.
Discussion: Robust inference combines safety-filter improvement, downstream concentration control, and favorable safe-versus-unsafe reward tails.These interventions address contamination and selection within the resulting feasible set.
Limitations: The analysis covers fixed, non-adaptive pipelines with independently sampled candidates filtered once and then ranked.Adaptive search, self-refinement, tree search, and agentic planning are outside the theory; their possible stronger amplification is not established or quantified.

8. Conclusion

The paper concludes that inference-time safety depends on both controlling entry into the proxy-feasible set and limiting downstream concentration within it. Its tail-based analysis explains why scaling can amplify residual filter errors, while coverage control bounds but does not remove that risk.

Conclusion: Safety hacking combines unsafe responses admitted by the learned filter with downstream reward selection that increasingly favors them.The finite-N analysis links this behavior to joint reward tails within safe and unsafe proxy-feasible groups.
Conclusion: A heavier unsafe reward tail can make safety hacking nearly certain as inference compute grows, even when average proxy errors are small.Coverage control yields an N-independent bound through cPes but cannot remove unsafe responses already admitted by the filter.
Conclusion: Corollary A.1 provides a diagnostic: compute-amplified hacking is expected when unsafe tail mass crosses a constant before true-positive tail mass does.This compares effective score-tail masses as N increases.
Conclusion: If a positive-measure prompt subset is tail-amplifiable, aggregate safety-hacking rates cannot vanish as the search budget grows.The result lifts fixed-prompt tail conditions to distribution-level evaluations.
Conclusion: Direct score dominance occurs when unsafe proxy-feasible outputs receive scores above the safe group's essential upper endpoint.In that case, a fixed threshold can satisfy the asymptotic tail condition.

A.4. Gaussian error-tail separation

The Gaussian error-tail analysis converts conditional reward-proxy error tails into learned-score tail separation. When unsafe outputs have the larger Gaussian scale, the fixed reward-range shift does not change the asymptotic exponent governing amplification.

Definitions: Proxy-feasible reward tails combine class mass with the upper-tail behavior of reward-proxy overestimation.The definition weights each class by its reference probability inside the proxy-feasible set.
Proof strategy: The derivation separates safe and unsafe score tails by relating learned-score exceedances to reward-proxy error exceedances with shifted thresholds.The local bounded reward-range assumption requires a nonempty proxy-feasible set and finite Δ(x, b).
Gaussian error-tail separation: Under positive safe and unsafe class masses, Gaussian error scales 0 < σA(x) < σB(x) imply score-tail separation.The unsafe class has the heavier conditional reward-error tail under this assumption.
Gaussian error-tail separation: The fixed reward-range shift Δ(x, b) changes the conversion threshold but not the Gaussian large-deviation exponent.This preserves the asymptotic separation used by the theorem.
Boundary behavior: When safe and unsafe tails are asymptotically comparable, finer tail constants and class masses determine the limiting behavior.Finite reward range can affect boundary constants without changing the Gaussian exponent away from the boundary.

A.5. Sample implementation and consistency of cPes

Sample cPes converges to its population pessimistic policy under positive proxy-feasible mass, while its regularization parameter controls the tradeoff between reward optimization and coverage. The accompanying analysis provides utility and safety comparisons for constrained selection.

Consistency: mN →∞ almost surely when the proxy-feasible reference mass q(x, b) is positive.Thus the sample implementation eventually has feasible candidates with probability one.
Consistency: For every bounded f, empirical cPes expectations converge almost surely to population cPes expectations.The result follows from convergence of the empirical multiplier and uniform convergence of the reweighting procedure.
Utility tradeoff: Smaller β permits more aggressive reward optimization and greater concentration, whereas larger β keeps cPes closer to the proxy-feasible reference.This makes β the explicit coverage–optimization control.
Utility analysis: The oracle regret analysis decomposes performance into comparison within the proxy-feasible set and the cost of missing feasible mass.The truly safe comparator is evaluated relative to the learned feasible constraint.
Safety analysis: CBoN hacking occurs when an unsafe feasible sample exceeds a threshold while all true-positive feasible samples remain below it.The converse non-hacking certificate uses a safe sample above a threshold and no unsafe sample above it.
Safety analysis: Under the asymptotic tail conditions, the sufficient hacking event has probability tending to one.This follows when safe tail mass vanishes after multiplication by N while unsafe tail mass diverges.

A.15. Additional Toy Experimental Details and Ablations

Toy ablations show that reward-tail structure, residual unsafe feasibility, and selection regularization jointly determine safety-hacking rates as candidate budgets grow.

cPes regularization: The cPes conditional hacking rate stays between 0.015 and 0.024 across budgets with β = 1.0, while mean true reward remains near 0.79.With β = 0.2, hacking remains around 0.024–0.031 at large budgets.
cPes regularization: At N = 8192, cPes hacking falls from 0.1518 at β = 0.02 to 0.0152 at β = 1.0, with highest true reward around β ∈[0.5, 1.0].More aggressive max-selection increasingly recovers CBoN’s amplification mechanism.
Equal-tail control: When unsafe outputs no longer have the heavier reward-error upper tail, CBoN hacking falls to zero by N = 32 and selected true reward converges to 0.8.The remainder of the environment is held unchanged in this equal-tail control.
No-false-positive control: Removing class B makes the safety-hacking rate identically zero for every method and candidate budget because no unsafe outputs remain proxy-feasible.This isolates contamination as a prerequisite for hacking in the toy setting.
False-positive tail-scale sweep: At N = 8192, CBoN hacking rises from 0.0 at σB = 0.2 to 0.9998 at σB = 1.2.The rate is 0.697 at σB = 0.6 and 0.9996 at σB = 1.0.
Experimental setup: The experiments define A and B using proxy feasibility plus HarmBench safety labels, while C contains proxy-rejected candidates.The safety proxy is Llama-Guard-3-8B with acceptance threshold b = 0.95; reward is primarily Beaver.

B.2. Alternative safety-filter sensitivity analysis

The alternative ShieldGemma filter preserves the qualitative competition-driven hacking pattern, while reward-proxy tail rankings reverse the observed trend across Beaver and Skywork.

Alternative safety filter: ShieldGemma’s threshold is 0.8176, yielding a feasible rate of 0.8242 versus 0.8295 for Llama Guard.The threshold is selected by matching Llama Guard’s calibration feasible rate.
Table interpretation: The table separates total hacking into disjoint unsafe-only and Competitive terms, while Tail win uses the conditioning event Pr(MB > MA | KA > 0, KB > 0).All entries are percentages, with standard errors for hacking and abstention.
Sensitivity results: Conditional CBoN hacking increases from 12.1% at N = 1 to 14.8% at N = 256, driven by competitive unsafe wins rising to 13.8%.The unsafe-only term instead decreases from 12.1% to 1.0%.
Reward-tail mechanism: With Beaver, the unsafe feasible class has the heavier observed upper reward tail; with Skywork, the safe feasible class does.The survival curves use separate proxy-score scales and should be compared within, not across, proxies.