PaperlayerView exact revision on arXiv

Source-linked AI summary

Risk Ceilings and Development Deadlines: Pacing AI under Uncertain Safety Productivity

Li Gan

arXiv:2610.11093v1econ.GNcs.CYq-fin.EC

TL;DR

The paper asks whether a regulator can promise both a risk ceiling and development deadline when safety productivity is unknown. It analyzes a learn-then-replay policy and finds that, in a calibrated state-risk case, it needs only 3.3 percent more productivity than the necessary bound.

  • Problem

    A regulator’s risk-ceiling and development-deadline promises may be incompatible when safety research productivity is unknown and weak research cannot be ruled out.

    Source: Introduction · pp. 1–2

  • Method

    The paper characterizes the productivity needed for both promises and evaluates learning first, then replaying full-speed development with spare compute in safety.

    Source: Introduction · p. 2, §III · p. 13

  • Results

    3.3 percent more productivity than the necessary bound suffices for learn-then-replay in the calibrated state-risk case.

    Source: Introduction · p. 2

  • Takeaways & Limitations

    The guarantee costs customer service value, while compute-allocation rules fix development dates rather than risk.

    Source: Introduction · p. 2

    The 3.3 percent productivity gap is specific to the calibrated state-risk case; with an even split between state and step risk, the gap is 21–72 percent.

    Source: Introduction · p. 2

Abstract

from arXiv · show

Can a regulator promise both a risk ceiling and a development deadline when safety productivity is unknown? A ceiling below the final model's unprotected hazard requires a minimum stock of safety knowledge, so both promises hold only if weak research can be ruled out. Learning first and then replaying development, with spare compute in safety, comes close to that minimum. In a calibration with only state risk, constant safety yield, and full knowledge transfer, it needs only 3.3 percent more productivity than the necessary bound. Customer services pay for the guarantee. Rules that fix compute allocation fix dates, not risk.

Introduction

The paper asks whether a regulator can guarantee both a risk ceiling and a development deadline when safety-research productivity is unknown. It identifies a minimum productivity threshold, a near-bound policy, and trade-offs in customer services and risk.

Introduction

  • A tight ceiling and deadline are compatible only if the productivity bound clears a threshold that rises with tighter ceilings and falls with longer deadlines.Every compliant path must build enough safety knowledge to reduce final-model hazard by a fixed log factor.Source: Introduction · p. 2
  • 3.3 percent more productivity than the necessary bound suffices for learn-then-replay in the calibrated state-risk case.The policy learns first, then replays full-speed development with spare compute allocated to safety; the gap is 21–72 percent with an even split between state and step risk.Source: Introduction · p. 2
  • The cheapest learn-then-replay policy learns for 7.5 months and gives up 4.1 percent of five-year service value, versus 5.0 percent with a full year of learning.Over two years it gives up 52 percent of service value.Source: Introduction · p. 2
  • Compute floors, pauses, and R&D-model lags fix development dates rather than risk, whose effect varies with safety productivity; a ceiling fixes hazard and lets delay adjust.At the scenario’s fitted yield, a ceiling using one year of lead requires 25 months at half that yield and 52 months at a quarter.Source: Introduction · p. 2

I. A Takeoff with Safety Research

The model studies compute allocation during a capability takeoff, with safety research building knowledge that reduces both operating and advancement hazards. It compares allocation rules by their effects on development, risk, and customer services.

I. A Takeoff with Safety Research

  • The baseline measures danger per month conditional on survival and ends hazard at completion; it does not model lab incentives to comply with the chosen rule.The takeoff runs from F0 = 0 to F1 = 1, with the endpoints corresponding to the Automated Coder and top-expert-dominating AI milestones.Source: §I · pp. 3–4
  • Safety knowledge is measured by its protection: adding ln 2 to S halves hazard, and safety yield is the monthly growth of that protection.The yield depends on capability and safety compute; with compute-dependent research, additional safety share has diminishing returns.Source: §I · pp. 3–4
  • State hazard comes from operating a model while capability stands still, while step hazard accrues as capability advances; safety knowledge scales down both.A pause removes step hazard while it lasts, but state hazard remains.Source: §I · p. 4
  • Compute grows exogenously, and capability and safety research progress according to compute shares and research-model capability.Without a lag, the research model tracks the frontier; an R&D-model lag instead has research use a model from ℓ months earlier.Source: §I · pp. 4–5
  • Customer services rise with capability and customer compute, while moving compute to safety or capability research reduces current services or delays service-quality improvements.Service loss is discounted value relative to full speed over a fixed horizon and does not weight services by survival probability.Source: §I · p. 4
  • The compared rules include compute floors, pauses, R&D-model lags, and a ceiling that slows development as needed and moves freed research compute to safety.A deadline limits the largest delay of any capability milestone relative to the full-speed path, not just the completion date.Source: §I · pp. 4–5

II. Which Commitments Can Be Guaranteed?

The section builds its answer in four steps, moving from valuing delay to identifying the knowledge requirement, assessing a simple policy, and examining date-fixing rules.

II. Which Commitments Can Be Guaranteed?

  • Subsection A prices a month of delay.Subsection B then shows the knowledge any compliant path needs.Source: §II · p. 5
  • Subsections C and D assess how close a simple policy comes to the minimum knowledge requirement.Source: §II · p. 5
  • Subsection E examines why rules that fix dates can raise risk.Source: §II · p. 5

A. The Value of a Month

A pause trades exposure to the current model against the safety knowledge that research builds. Under a repeated-path benchmark, its value depends on the remaining hazard and research productivity, with timing effects that can change under other conditions.

A. The Value of a Month

  • A pause adds exposure at rate w(t) while safety research builds knowledge at rate m; the pause’s safety allocation can raise that learning benefit.Knowledge reduces hazard during the pause and along the resumed path.Source: A. The Value of a Month · pp. 5–6
  • A pause of δ months lowers cumulative hazard under the repeated-path benchmark if and only if v_sp(t) > 0 when m > 0; at m = 0, it adds δw(t).For a finite pause, the hazard saving accounts for knowledge accumulating during the pause and protecting the resumed path.Source: A. The Value of a Month · p. 6
  • With constant yield and rising unprotected state hazard, earlier delay is more valuable, but rising safety yield can make a later pause more productive.Growing compute also breaks the repeated-path benchmark because development can catch up after a pause.Source: A. The Value of a Month · p. 6

B. A Necessary Knowledge Requirement

A tight ceiling requires a minimum stock of safety knowledge by the end of takeoff, so a joint ceiling and deadline guarantee requires productivity above a threshold. The necessary bound is generous because it assumes maximal safety learning throughout the available time.

B. A Necessary Knowledge Requirement

  • The required knowledge is ln[h(1)/¯c], while maximal monthly learning is bounded by a¯σmax¯y, giving a minimum learning-time test for any policy.In the half-state-risk scenario, at least ln 57 ≈ 4.05 knowledge is needed; at the fitted yield, compliance takes at least 17 months.Source: B. A Necessary Knowledge Requirement · p. 7
  • 17 months of minimum learning fits within the 30-month horizon at the fitted yield, but not below 17/30 ≈ 0.57 of that yield.The cutoff is a necessary productivity bound for the scenario, not a guarantee that a feasible policy exists.Source: B. A Necessary Knowledge Requirement · p. 7
  • The condition is necessary but not sufficient because it allows maximal learning with all compute assigned to safety, potentially leaving none to advance capability.A lower ceiling raises the knowledge requirement, while a longer deadline or higher productivity relaxes it.Source: B. A Necessary Knowledge Requirement · p. 7
  • Prescribed allocation rules fix arrival dates independently of safety productivity, whereas a ceiling fixes hazard and allows delay to adjust.The bound concerns the stock required for the final model’s state hazard to fall below the ceiling.Source: B. A Necessary Knowledge Requirement · pp. 7–8

C. Learning First, Then Replaying

Learn-then-replay first builds safety knowledge at the initial capability level, then follows the reference development path with that protection in place. Spare compute during replay can be assigned to additional safety research, and the replay condition specifies when this policy class can meet both promises.

C. Learning First, Then Replaying

  • Learn-then-replay devotes all compute to safety first, then reproduces the reference capability path with the additional protection already built.Because compute grows, replaying the path requires only the reduced research share x0e^-Gτ, leaving the remainder for safety or customers.Source: C. Learning First, Then Replaying · p. 8
  • The guarantee test checks initial learning and replay separately, with the worst productivity amin determining the largest replay hazard.If the full allowed lead L cannot reduce the peak enough, a shorter learning phase with the same replay share cannot do so either.Source: C. Learning First, Then Replaying · pp. 8–9
  • Within the replay class, condition (7) is exact: some allowed learning time and replay safety share meet the ceiling and deadline for every a ≥ amin if and only if it holds.The smallest productivity bound satisfying the condition for a given replay safety share is that share’s replay bound.Source: C. Learning First, Then Replaying · pp. 8–9
  • The replay safety share trades customer compute for safety learning, with spare-compute replay assigning all compute not needed for development to safety.Usual-share replay provides services after its initial learning phase; spare-compute replay provides none until replay ends, 30 months into the scenario.Source: C. Learning First, Then Replaying · p. 9

D. How Close the Bounds Are

With only state risk, constant safety yield, and knowledge that remains useful, the spare-compute replay bound is close to the necessary productivity bound. The gap widens when knowledge yield changes or step risk is included, while the guarantee entails losses in customer services.

D. How Close the Bounds Are

  • With all risk from frontier state and constant yield, one year of compute growth cuts replay’s research share to 12.5%; replay learns at 2.59 versus necessary-bound 2.74, a 3.3% gap.The bounds’ ratio is 2.74 × 30/(2.74 × 12 + 2.59 × 18) ≈ 1.033; the threshold lies between them.Source: D. How Close the Bounds Are · pp. 9–10
  • 2.5 is the factor when safety yield triples from the first to last milestone, and 1.8 when half of knowledge is lost as capability rises.These factors bound the replay policy’s shortfall because the necessary bound is loose in both cases; the result does not show learning first is best when yield rises.Source: D. How Close the Bounds Are · p. 10
  • 21–72% is how far the spare-compute replay bound exceeds the necessary bound with an even split between state and step risk, across the stated ceiling range.The comparison covers ceilings between the scenario’s ceiling and 1 a month.Source: D. How Close the Bounds Are · p. 10
  • At the scenario ceiling of 0.0492 a month, with a year of learning, spare-compute replay needs a 0.69 productivity bound and gives up 13 percent of five-year service value and all two-year value; usual-share replay needs 1.08 and gives up 4.7 percent.Source: D. How Close the Bounds Are · p. 10
  • 7.5 months of learning minimizes five-year service loss within the replay class at fitted yield, giving up 4.1% versus 5.0% for a full year of learning.The optimized policy then assigns 21.0% to research, 40.1% to safety, and 38.9% to customers; these costs are not minimized over all compliance paths.Source: D. How Close the Bounds Are · p. 10

E. Exposure When Safety Research Is Ineffective

When safety research is ineffective, fixed compute allocations can increase cumulative takeoff hazard by keeping the lab longer at each capability level. Whether a pause lowers risk depends on how later faster progress changes exposure to hazardous models.

E. Exposure When Safety Research Is Ineffective

  • Longer time at each capability level raises state-hazard exposure when safety work produces no protection, while step hazard does not fall if the same total capability increase occurs.In the scenario’s floors, milestone timing stretches by about 2.5 on the scale e^γt − 1 of research-capacity growth.Source: E. Exposure When Safety Research Is… · pp. 10–11
  • A constant research share below the full-speed share increases cumulative takeoff hazard when safety research is ineffective, and the strict increase persists for sufficiently small positive productivity.The result assumes R = F and h positive on a capability interval.Source: E. Exposure When Safety Research Is… · p. 11
  • 4.9 points lower catastrophe probability is the scenario’s result for an early pause when safety research does nothing; a pause started at month 15 raises it by 7.7 points.The floors and R&D-model lag raise the probability by 5.4 and 16.8 points, respectively.Source: E. Exposure When Safety Research Is… · p. 12
  • With ineffective research, a pause lowers risk only if later saved state exposure exceeds the hazard accumulated while waiting.An early pause may be offset by faster progress through hazardous levels later; a late pause leaves fewer subsequent levels over which to compensate.Source: E. Exposure When Safety Research Is… · p. 12

III. A Calibrated Scenario

The calibrated scenario compares fixed policy rules across safety-yield settings using an illustrative worked example, reporting their delay, risk, and customer-service trade-offs. Results also vary with how hazard primitives are held fixed and with assumptions about knowledge transfer and hazard shape.

Calibration

  • The worked example gives an 18-month takeoff and 75% catastrophe probability, which a safety-case regime could cut to 40%; its scenario’s deadline is 12 months of maximum milestone delay.The example’s magnitudes are judgments, not estimates.Source: §III · pp. 12–13

Replay policy

  • The replay search chooses when learning stops and the smallest safety share that meets the final-hazard knowledge requirement, balancing initial protection against delayed customer services.The condition compares end-of-replay knowledge at the worst yield with the log reduction required by final hazard; it solves the replay problem, not unrestricted control.Source: §III · p. 13

Service costs

  • The smaller five-year service-value loss does not erase the early shortfall: after replay, services resume at full-speed shares, enlarging the five-year denominator.Most of the two-year horizon is spent learning or developing the delayed frontier.Source: §III · p. 13

Comparison scope

  • Figure 2 rescales hazard primitives at each yield to preserve 75% full-speed probability, so it compares matched-risk versions rather than varying research productivity alone.The fitted yield and hazard shape are illustrative, and the bounds’ distance also depends on compute growth, the deadline, and returns to safety compute.Source: §III · p. 13, Figure 2 · p. 14

Yield sweep

  • At a quarter of fitted yield, the ceiling raises catastrophe probability by 20.1 points, while floors lower it by 15.3 and pause-with-safety by 20.5.The ceiling uses 52 months of lead at a quarter yield and 25 months at half yield; Figure 2 holds settings fixed while scaling yield.Source: §III · p. 13, Figure 2 · p. 14

Fixed hazard primitives

  • With hazard primitives fixed, halving yield raises full-speed catastrophe probability to 93.5%; the ceiling lowers it to 88.1%, a 5.4-point reduction, using 32.8 months of lead.At one hundredth of fitted yield, floors instead raise risk to 79.6%, above full speed’s 75%, while retaining a year of lead.Source: §III · p. 13

Technology-version comparisons

  • Across ten technology versions, floors lower catastrophe probability by 15–71 points and pause-with-safety by 21–68, while ceiling effects range from a 24-point rise to a 74-point fall.Late delay pays when safety research grows more productive with capability or compute, or danger continues after takeoff, because late knowledge protects the whole tail.Source: §III · p. 14

IV. Policy Commitments and Their Scope

The bounds offer a pre-commitment test: whether a regulator can jointly promise a ceiling and deadline depends on the productivity it can rule out. The analysis also qualifies what ceilings and service budgets guarantee and identifies idealizations that limit the results.

A test for joint promises

  • If the necessary condition fails at the productivity bound, no compliance strategy keeps both promises; if the replay condition holds, learn-then-replay does so at the service cost.Between the conditions, the band is narrow with state risk and persistent knowledge, and widens with step risk, rising yields, or lost knowledge.Source: §IV · p. 14

Evidence and affordability

  • The productivity bound is the binding input: evidence ruling out yields below the replay bound makes a universal promise conditionally feasible, while evidence raising the bound lowers its price.Measuring hazard reduction and knowledge transfer helps determine whether a joint promise is affordable.Source: §IV · p. 15

What budgets guarantee

  • At the scenario ceiling, the monthly ceiling implies a catastrophe probability of 77.2%, above full speed’s 75%; a cumulative-hazard budget bounds catastrophe probability directly.A five-year percentage service budget permits 13.65 times the absolute loss of a two-year budget stated at the same percentage.Source: §IV · p. 15

Scope and assumptions

  • The model assumes exogenous compute growth, idealized enforcement and hazard measurement, a fixed-delay deadline, and full transfer of early-model knowledge to later models.If knowledge does not transfer fully, learn-then-replay’s advantage shrinks; scenario magnitudes are judgments rather than estimates, though Propositions 1–5 do not depend on them.Source: §IV · p. 15

V. Conclusion

The conclusion says a tight ceiling and deadline require ruling out weak safety research; a necessary bound and learn-then-replay bracket the productivity needed. In the stated calibration, the bounds are close, while weaker research requires more customer compute.

V. Conclusion

  • With constant yield, only state risk, and full knowledge transfer, learn-then-replay needs 3.3% more productivity than the necessary bound in the calibration.The necessary bound and policy bracket the productivity required for the joint promise.Source: §V · p. 15
  • Rules that fix compute allocation fix dates instead; with ineffective research, their effect on risk depends on timing.Source: §V · p. 15
  • The weaker the research a regulator must allow for, the more customer compute a joint promise consumes.Source: §V · p. 15

A. Calibration and Implementations

The appendix specifies the calibration and numerical rules, then examines how the knowledge bound and the value of waiting change under alternative assumptions. Its extensions show that low safety productivity can make a tight ceiling infeasible by a fixed deadline, while rising yield can change the value of waiting at later capabilities.

A. Calibration and Implementations

  • The baseline yield is σ_b = 0.086116 per month, or 8.3 percent hazard removed in a benchmark month.The baseline uses constant yield and fits a 12-month pause at F = 0 with the compute clock held fixed, safety share s0, and the later reference path repeated.Source: A. Calibration and Implementations · p. 19
  • With spillovers, safety compute can advance capability during a pause even when dedicated research is zero; dates remain invariant to the safety-yield scale when controls stay fixed.The ceiling controller is not cost-minimizing compliance, and zero pace cannot instantly restore feasibility if initial state hazard exceeds the ceiling.Source: A. Calibration and Implementations · pp. 19–20
  • 13.65 times as much absolute discounted service loss fits a five-year percentage budget as a two-year one; a fixed absolute budget does not preserve that percentage across horizons.Under the stated closure, service value grows after completion, making these finite-horizon budgets.Source: A. Calibration and Implementations · p. 20
  • Scenario judgments supply assumptions rather than estimated primitives, and matching the 18-month, 75 percent, and 40 percent targets is internal consistency, not independent intervention evidence.The mapping from a safety-case regime to a clock-stopped pause is also a model assumption.Source: A. Calibration and Implementations · p. 20

B. Extensions of the Value of a Month and the Knowledge Bound

  • A higher yield at later capabilities can increase waiting’s benefit through σ′(F)A, but only if it exceeds exposure and step-hazard terms.This is a repeated-path marginal-value result, not a policy comparison when growing compute changes the resumed path.Source: B. Extensions of the Value of a Month… · p. 23
  • With a fixed deadline, sufficiently low productivity makes a ceiling below h(1) infeasible; milestone knowledge gates are necessary but do not establish feasibility.The milestone gates relax joint compute use for learning and development and ignore nonnegative step hazard.Source: B. Extensions of the Value of a Month… · pp. 23–24

C. Additional Feasibility and Sensitivity Checks

The checks show that prescribed rules’ risk effects vary with timing, productivity, and hazard convention, alongside tests of replay policies and finite-set commitments.

C. Additional Feasibility and Sensitivity Checks

  • Across ten safety technologies, compute floors, pauses, and safety floors lower risk in every version, while very weak-yield floors can leave fixed dates alongside rising risk.The ceiling and R&D-model lag raise risk in several versions; the very weak yields illustrate guarantee scope, not estimated plausible productivity.Source: C. Additional Feasibility and… · p. 24
  • At half yield, the fixed ceiling lowers risk by 5.4 points with fixed primitives but raises it under 75 percent normalization; both cases miss the deadline.Source: C. Additional Feasibility and… · p. 24
  • In the replay simulation, a 3.70-month initial safety period keeps hazard at the 0.500 ceiling, with service losses of 13.36 percent over two years and 0.98 percent over five.The five-year 5 percent service budget is met for every a ≥ 1 in this example; Proposition 3 does not itself impose a service budget.Source: C. Additional Feasibility and… · p. 27
  • When half of early knowledge is lost, replay needs more than the fitted yield, so the necessary bound no longer closely tracks its true productivity threshold.The necessary bound assumes maximum-yield learning and ignores decay, making it loose in both tested variants.Source: C. Additional Feasibility and… · p. 28
  • A finite-set commitment keeps sampled takeoff hazard at 0.421 below the 0.50 ceiling, but fails the five-year 5 percent and every two-year service budget.It meets the lead and five-year 15 and 30 percent budgets, but is not an optimal controller or guarantee for arbitrarily small yields.Source: C. Additional Feasibility and… · p. 28

D. Numerical Methods

  • Numerical checks use bisection and a 4,001-point replay grid for fixed-timing productivity bounds, while the service-loss search checks multiple timings and grid resolutions.The methods also check risk increases, service budgets, and pointwise risk improvement across the tested environments.Source: D. Numerical Methods · p. 29

E. Notation

Loading 2610.11093v1…