PaperlayerView exact revision on arXiv

Source-linked AI summary

Training Compute Thresholds: Features and Functions in AI Regulation

Lennart Heim, Leonie Koessler

arXiv:2405.10799v2cs.CYcs.LG

TL;DR

Regulators need an early way to identify GPAI models that may create large-scale societal risks, while training compute remains an imperfect risk proxy. The paper evaluates compute thresholds and argues they are currently the best initial filter for oversight and further scrutiny, with other metrics informing mitigation measures. It concludes that thresholds should evolve and be complemented as technology and market conditions change.

  • Problem

    Regulators need a measurable way to identify GPAI models that may pose risks of large-scale societal harm, but training compute is only a crude proxy for risk.

    Source: Executive Summary · p. 2, §4 · p. 12

  • Method

    The paper analyzes training compute’s regulatory features and limitations, its existing use in US and EU rules, and the functions compute thresholds should serve.

    Source: Executive Summary · pp. 2–3, §1 · p. 6

  • Results

    Training compute is currently the best available tool for identifying potentially risky GPAI models and triggering regulatory oversight and further scrutiny.

    Source: Executive Summary · p. 4, §7 · p. 28

  • Takeaways & Limitations

    Use compute thresholds as an initial filter, then apply model evaluations, risk assessments, and other metrics to determine appropriate mitigation measures.

    Source: Executive Summary · p. 4

    Compute thresholds can be overinclusive or underinclusive because risk varies across models and algorithmic efficiency changes the compute-risk relationship over time.

    Source: §4 · p. 12, §6.4.3 · p. 24

Abstract

from arXiv · show

Regulators in the US and EU are using thresholds based on training compute--the number of computational operations used in training--to identify general-purpose artificial intelligence (GPAI) models that may pose risks of large-scale societal harm. We argue that training compute currently is the most suitable metric to identify GPAI models that deserve regulatory oversight and further scrutiny. Training compute correlates with model capabilities and risks, is quantifiable, can be measured early in the AI lifecycle, and can be verified by external actors, among other advantageous features. These features make compute thresholds considerably more suitable than other proposed metrics to serve as an initial filter to trigger additional regulatory requirements and scrutiny. However, training compute is an imperfect proxy for risk. As such, compute thresholds should not be used in isolation to determine appropriate mitigation measures. Instead, they should be used to detect potentially risky GPAI models that warrant regulatory oversight, such as through notification requirements, and further scrutiny, such as via model evaluations and risk assessments, the results of which may inform which mitigation measures are appropriate. In fact, this appears largely consistent with how compute thresholds are used today. As GPAI technology and market structures evolve, regulators should update compute thresholds and complement them with other metrics into regulatory review processes.

Executive Summary

Training compute is a useful, measurable basis for initially identifying potentially risky GPAI models, but it is too crude to determine mitigation measures alone. The paper therefore supports using compute thresholds to trigger oversight and further scrutiny, complemented by capability evaluations and other metrics.

Features of training compute

  • Training compute correlates with model loss, capabilities, and risks, while remaining quantifiable, externally verifiable, and measurable before development or deployment.It is also relatively resistant to circumvention and tracks training-resource costs.Source: Executive Summary · pp. 2–3

Limitations of training compute

  • Training compute is an imperfect risk proxy because high-compute models may pose limited concerns, low-compute models may pose significant risks, and algorithmic efficiency can weaken the relationship over time.The compute-risk relationship may shift gradually as the field evolves.Source: Executive Summary · p. 3

Functions of compute thresholds

  • Compute thresholds should initially identify GPAI models warranting regulatory oversight and further scrutiny, including visibility into risks and company investment in understanding them.Precautionary mitigations may be applied initially but should later be removable if unnecessary.Source: Executive Summary · p. 3
  • Compute thresholds should not alone determine mitigation measures; capability evaluations and other, more precise metrics can inform those decisions within a broader regulatory framework.Most AI requirements should not hinge on training compute alone.Source: Executive Summary · p. 3, Figure 2 · p. 4

Current regulatory use

  • The US AI EO uses 10^26 operations for notification and related reporting, while the EU AI Act uses 10^25 operations to trigger notification, evaluations, systemic-risk mitigation, incident reporting, and cybersecurity requirements.These regimes largely use compute thresholds as initial triggers for oversight and scrutiny.Source: Figure 1 · p. 2, Executive Summary · p. 3

Challenges and evolution

  • Thresholds should be updated as risks, algorithmic efficiency, market participation, the threat landscape, and societal vulnerability change.Efficiency may justify lowering thresholds, while limited risks or burdensome oversight may justify raising them.Source: Executive Summary · p. 4
  • Other metrics can better track risk, but their measurement difficulty makes training compute especially suitable for an early lifecycle filter.Potential combinations include compute with capability evaluations or user counts, subject to when those metrics become measurable.Source: Executive Summary · p. 4

Conclusion

  • Overall, training compute is currently the best available tool for identifying potentially risky GPAI models and triggering proactive regulatory oversight and further scrutiny.It can complement targeted capability evaluations that ultimately inform mitigation measures.Source: Executive Summary · p. 4

1 Introduction

GPAI models are becoming more capable and widely relied upon, increasing the potential for large-scale societal harm and motivating compute-based regulation. The paper examines which features make training compute useful for regulation and what functions compute thresholds should serve.

Motivation

  • GPAI models use increasingly large training runs, and higher capabilities can increase risks through dual-use applications, wider use, and greater reliance.Examples include automated hacking and biased outputs.Source: §1 · p. 6

Existing regulation

  • The US and EU already use training-compute thresholds to trigger notification, red-teaming, model evaluations, systemic-risk mitigation, incident reporting, and cybersecurity requirements.The US threshold is 10^26 operations, while the EU threshold is 10^25 floating-point operations.Source: §1 · p. 6

Research gap

  • Academic treatment of compute thresholds remains scarce despite increasing regulatory reliance, with prior work leaving features, regulatory functions, and existing rules insufficiently addressed.The paper positions its analysis as a response to this gap.Source: §1 · p. 6

Research questions

  • The paper asks which features of training compute matter for GPAI regulation and what function compute thresholds should have in regulatory systems.It analyzes the concept, useful features, limitations, regulatory functions, and implementation challenges.Source: §1 · p. 6

Scope and terminology

  • The paper focuses on computational resources used during model training, distinguishing training compute thresholds from infrastructure-capacity thresholds.Its terminology emphasizes the training phase rather than mere possession of computing infrastructure.Source: §1 · p. 6

2 Training Compute

Training compute is the total number of operations used to train an AI model, but the paper recommends measuring only pre-training compute for regulatory thresholds. This focus reflects empirical scaling laws and avoids the practical burden of repeatedly measuring post-training enhancements.

2 Training Compute

  • Training compute is the total number of operations executed throughout an AI model’s training process, distinct from processing performance measured per second.The paper broadly includes integer, floating-point, and other operations, while noting that current AI training predominantly uses floating-point operations.Source: §2 · p. 7
  • Since 2010, AI training compute has grown by about 4× per year, alongside increases in training data and model parameters.This growth is substantially faster than the approximately two-year doubling associated with the pre-deep-learning era.Source: §2 · p. 7, Figure 3 · p. 8
  • The paper recommends measuring pre-training compute rather than adding compute from fine-tuning, RLHF, and other post-training enhancements.Post-training enhancements can substantially improve capabilities, but existing scaling laws describe performance primarily as a function of pre-training compute.Source: §2 · p. 7, Figure 4 · p. 9
  • Including post-training compute is impractical because fine-tuning is repeated across versions and often performed by downstream developers.Re-measuring every enhanced version would burden regulatees and expand the number of entities requiring evaluation.Source: §2 · pp. 7–8
  • Capability evaluations should account for post-training enhancements, while minor variations may not require repeating evaluations when the pre-trained model has already been assessed.The paper recommends periodic re-evaluation of enhanced versions and re-evaluation when they constitute a new model family.Source: §2 · pp. 8–9

3 Features of Training Compute Useful for GPAI Regulation

Training compute is useful for GPAI regulation because it tracks capabilities and risks while remaining measurable, difficult to circumvent, and observable before deployment. It also broadly tracks development resources, though falling compute costs may expand regulation to smaller actors over time.

3 Features of Training Compute Useful for GPAI Regulation

  • Training compute correlates with loss, capabilities, and potential risks, although the direction and balance of some capability-related risks remain empirically uncertain.Scaling laws connect compute to loss, and loss improvements tend to correlate with downstream capabilities; higher capabilities can increase misuse and reliance risks, while some failure risks may decrease.Source: §3 · p. 9
  • Training compute is relatively easy to calculate from model specifications or hardware-use data, and its unidimensional form is more durable than many alternative metrics.The paper contrasts it with multidimensional measures such as data quality and potentially outdated capability benchmarks.Source: §3 · pp. 9–10
  • Training compute is difficult to circumvent because reducing it for fixed architectures and algorithms generally lowers capabilities and associated risks.Unlike some benchmark-based metrics, compute cannot usually be reduced while preserving the same capability level; algorithmic efficiency remains a separate threshold-setting challenge.Source: §3 · p. 10
  • Compute can be estimated before development and calculated before deployment, enabling compute-indexed precautions during intensive training runs.Companies plan large training runs around substantial GPU and financial requirements, creating incentives to estimate compute for resource allocation and security planning.Source: §3 · p. 10
  • External parties, especially compute providers, can verify compute usage without accessing proprietary model architectures or training data.This supports confidentiality-preserving compliance verification.Source: §3 · p. 10
  • Training cost scales with compute, so thresholds can focus regulatory burdens on well-resourced organizations rather than smaller actors.The paper gives an illustrative comparison in which 10^26 operations costing $70M are approximately ten times more expensive than 10^25 operations costing $7M.Source: §3 · pp. 10–11, Figure 5 · p. 11
  • Falling computational prices may let more actors cross fixed compute thresholds, expanding regulatory scope and complicating oversight.Observed FLOP-per-dollar performance improvements could increase burdens for smaller organizations over time.Source: §3 · p. 11

4 Limitations of Training Compute Relevant for GPAI Regulation

Training compute is only a crude proxy for risk because the links from compute to loss, capabilities, and risk are imperfect and context-dependent. Its usefulness may decline as algorithmic progress or shifts away from deep learning weaken the assumptions behind scaling laws.

4 Limitations of Training Compute Relevant for GPAI Regulation

  • The relationship between training compute and capabilities can be non-smooth, with inverse scaling on some tasks and apparent capability jumps despite gradual loss improvements.Some inverse-scaling relationships have re-emerged with further scaling, so compute does not imply uniformly increasing capability across tasks.Source: §4 · p. 12
  • Capability increases do not translate smoothly into risk because risk depends on context, including offense-defense balance, threat actors, and societal vulnerability.These factors can materially affect risks beyond a model’s capabilities alone.Source: §4 · p. 12
  • Training compute is therefore only a very crude risk proxy: it correlates with loss, loss with capabilities, and capabilities with risk, but none of these correlations is perfect.The proxy relationship can weaken when capability or risk changes are discontinuous or shaped by factors outside the model.Source: §4 · p. 12
  • Training compute’s regulatory value depends on scaling laws continuing to hold, but paradigm shifts or reduced importance of pre-training could eventually undermine that relationship.The paper expects any deterioration to occur gradually rather than as an abrupt disruption.Source: §4 · p. 12

5 Functions of Training Compute Thresholds in GPAI Regulation

Compute thresholds should function primarily as an initial filter for regulatory oversight and further scrutiny, not as the sole determinant of mitigation measures. The US AI EO and EU AI Act largely reflect this approach through reporting, evaluation, and risk-assessment requirements.

5.1 Initial Filter

  • Training compute is an easily measurable and externally verifiable proxy that can reduce compliance and enforcement costs while focusing attention on potentially high-risk GPAI models.Its use as a filter avoids imposing intensive scrutiny on the much larger fraction of models unlikely to pose large-scale societal harm.Source: §5.1 · p. 13
  • Compute thresholds should identify models warranting regulatory oversight and further scrutiny, while capability thresholds and risk assessments should inform specific mitigation measures.Compute is a crude proxy for risk, so models crossing a threshold should undergo more rigorous analysis before definitive measures are required.Source: §5 · pp. 12–13, §5.1 · p. 13
  • Most AI requirements should not hinge on training compute because application-layer risks, compute-independent risks, and ex post harms require other regulatory criteria.Bias, discrimination, fairness, copyright, privacy, tort, and criminal-law concerns are identified as examples requiring application-specific or generally applicable rules.Source: §5.1 · pp. 13–14

5.2 US AI Executive Order

  • The US AI EO requires notification and reporting for models above 10^26 operations, or 10^23 operations when trained primarily on biological sequence data.The reporting covers development activities, model-weight security, red-teaming results, and related mitigation measures; as of July 2024, no model had officially reached the general threshold.Source: §5.2 · p. 15, Figure 7 · p. 15
  • The US AI EO uses compute thresholds mainly to increase regulatory oversight and scrutiny rather than directly mandate particular security or red-teaming measures.The paper argues that providers should be able to relax precautionary security measures if evaluations establish sufficiently safe capabilities.Source: §5.2 · pp. 15–16

5.3 EU AI Act

  • The EU AI Act presumes systemic risk above 10^25 floating-point operations and imposes notification, evaluation, risk-assessment, incident-reporting, and cybersecurity requirements.The compute threshold is one pathway among several, but currently the most concretely specified route to systemic-risk classification.Source: §5.3 · p. 16, Figure 8 · p. 17
  • The EU AI Act’s codes of practice can differentiate mitigation measures based on model-evaluation results rather than leaving them solely determined by compute thresholds.The paper therefore interprets the Act as broadly embedding compute thresholds within a staged review framework.Source: §5.3 · pp. 17–18
  • The EU AI Act allows providers to demonstrate that a model does not pose systemic risk despite crossing the compute threshold, potentially ending further evaluations and precautionary measures.This exception reflects the threshold’s role as a presumption and screening mechanism rather than a conclusive risk determination.Source: §5.3 · p. 18

6 Challenges for Training Compute Thresholds in GPAI Regulation

The paper identifies threshold-setting, domain specificity, updating over time, and complementary metrics as challenges for regulating with training compute.

6 Challenges for Training Compute Thresholds in GPAI Regulation

  • Regulators must determine where to set compute thresholds at a given point in time and whether domain-specific thresholds are appropriate.The paper also examines how thresholds should be updated as technology and markets evolve.Source: §6 · p. 18
  • Regulatory review may need metrics beyond training compute, including risk estimates, model capability evaluations, and effective compute.These metrics are considered as ways to distinguish more- and less-risky models when compute alone is insufficient.Source: §6 · p. 18
  • The appropriate threshold and complementary metrics must be reconsidered as GPAI technology and market structures change.The paper treats threshold updating as an ongoing regulatory design problem rather than a one-time choice.Source: §6 · p. 18

6.1 Where to Set Compute Thresholds

Compute thresholds can target models above, at, or below the capability frontier, trading off focus, safety margin, and regulatory burden. The US currently targets models above the frontier, while the EU targets models at and above it through a moving threshold.

Where to Set Compute Thresholds

  • Models above the frontier focus oversight on the few models most likely to be risky but may miss other risky models.These models are currently described as requiring roughly 10^26–10^27 operations and may have unprecedented, difficult-to-predict capabilities.Source: §6.1 · pp. 18–19
  • Models at the frontier may already show warning signs of dangerous capabilities, supporting evaluations and risk assessments even when developers report no such capabilities.This tier is currently described as roughly 10^25–10^26 operations.Source: §6.1 · p. 19
  • Models below the frontier provide a larger safety margin but risk imposing burdens on companies and regulators whose models later prove non-risky.This cautious tier is currently described as roughly 10^24–10^25 operations and could become warranted if society fails to keep pace with rising capabilities.Source: §6.1 · p. 19
  • The EU AI Act targets GPAI models at and above the current capability frontier, while the US AI EO currently targets models above it.The EU approach triggers systemic-risk requirements, including notification, evaluations, risk assessments, incident reporting, mitigation, and cybersecurity measures.Source: §6.1 · pp. 19–20, Figure 10 · p. 19

Post-Training Enhancements

Compute thresholds should reflect the capabilities of post-trained systems rather than only pre-trained models. Future enhancements may justify a safety buffer, but pre-training compute is expected to remain critical to relative capability.

Post-Training Enhancements

  • Thresholds should be based on the capabilities of enhanced systems such as ChatGPT, not solely on their pre-trained models.The relevant regulatory question concerns capabilities after post-training enhancements, because those are the capabilities systems ultimately obtain.Source: Post-Training Enhancements · p. 20
  • Regulators could add an order-of-magnitude safety buffer in equivalent pre-training compute for unknown future post-training improvements.Better enhancements could make models trained with the same pre-training compute display more dangerous capabilities over time.Source: Post-Training Enhancements · p. 20
  • Post-training enhancements may raise absolute capabilities but are unlikely to let models skip multiple generations of compute scaling.Their effectiveness can also depend on larger models, leaving pre-training compute critical to relative capabilities.Source: Post-Training Enhancements · p. 20

6.2 Domain-Specific Compute Thresholds

Domain-specific compute thresholds are appropriate only where scaling laws connect training compute to risk and the threshold does not bring too many actors into scope. The paper remains uncertain about biology, especially for specialized biological tools.

6.2 Domain-Specific Compute Thresholds

  • Domain-specific thresholds make sense only when scaling laws strongly relate training compute to risk and the threshold leaves a practicable number of actors in scope.Both conditions can vary substantially across domains.Source: §6.2 · p. 20
  • The paper is uncertain whether compute thresholds suit AI-enabled biological tools because scaling-law evidence is limited and data quality may matter more than compute.These tools generally require about 100× less training compute than GPAI models, potentially bringing many actors into scope.Source: §6.2 · p. 21
  • Compute thresholds could make sense for biology-focused LLMs, but available scaling-law research remains scarce.The paper distinguishes these LLMs from specialized biological tools because they support varied applications rather than one narrowly defined biological task.Source: §6.2 · p. 21

6.3 Why, When, and How to Update Compute Thresholds

Compute thresholds should remain updateable as GPAI capabilities, risks, technology, and market access evolve. Updating them requires balancing risk correspondence against regulatory and oversight burdens, while considering both fixed and dynamic designs.

6.3 Why, When, and How to Update Compute Thresholds

  • Compute thresholds should be updated as understanding of risky models, algorithmic efficiency, price-performance, threat conditions, and societal adaptation change.The direction of adjustment may be upward or downward depending on how these developments affect risk and regulatory scope.Source: §6.3 · pp. 21–22
  • The US and EU use fixed thresholds that regulators update over time, while alternatives could tie thresholds dynamically to the highest-compute model.Even dynamic thresholds should remain adjustable for improved risk understanding and other relevant developments.Source: §6.3 · p. 21
  • If precautionary regulation aims to detect when models reach a given risk level, thresholds may move upward as understanding of capabilities and societal risks improves.Current requirements are imposed amid uncertainty because models might pose high risks, rather than because regulators are certain that they do.Source: §6.3 · p. 21
  • Algorithmic efficiency creates a regulatory dilemma: lowering thresholds preserves capability correspondence, while raising them can limit burdens as more actors reach the same capabilities.Efficiency improvements reduce the compute needed for models with a given capability level but can expand the regulated population.Source: §6.3 · p. 22
  • Hardware price-performance improvements do not change the risk associated with a given compute threshold but can increase the number of actors able to cross it.Raising thresholds may reduce oversight costs while accepting higher societal risk.Source: §6.3 · pp. 22–23
  • The US framework calls for regular updates, with examinations at least every couple of months and potentially more often as efficiency or price-performance changes accelerate.The EU framework refers to technological developments such as algorithmic improvements and increased hardware efficiency but omits improved risk understanding and societal developments.Source: §6.3 · pp. 21–23

6.4 Alternative Metrics

The paper compares risk estimates, capability evaluations, effective compute, and other development or deployment metrics with training compute. These alternatives can add information about risk, but often sacrifice early measurability, external verifiability, standardization, or resistance to circumvention.

6.4.1 Risk Estimates

  • Risk estimates directly target societal harm but are immature and effort-intensive, so they should inform mitigation decisions after compute thresholds rather than replace them as an initial filter.Source: §6.4.1 · p. 23

6.4.2 Model Capability Evaluations

  • Capability evaluations provide more risk-relevant information than training compute, but their difficulty, prompt sensitivity, lack of standardization, gaming risks, and late availability limit their use as first-pass thresholds.Source: §6.4.2 · pp. 23–24

6.4.3 Effective Compute

  • Effective compute adjusts training compute for algorithmic efficiency by measuring equivalent compute relative to a specified model-performance metric and reference point.Source: §6.4.3 · p. 24
  • Technical performance metrics scale more smoothly and are easier to verify, whereas capability benchmarks are closer to real-world impact but can scale suddenly and reduce risk to specific tests.Source: Figure 11 · p. 25, §6.4.3 · p. 25
  • Effective compute is promising for internal company use but currently unsuitable for regulation because measurement standards, lifecycle timing, external access, and technical expertise remain inadequate.Source: §6.4.3 · pp. 25–26

6.4.4 Parameters, Data, and Other Metrics

  • Parameters and training-data quantity are quantifiable but largely captured indirectly by compute, while architecture, data quality, users, applications, and harm have distinct measurement or timing limitations.Source: §6.4.4 · pp. 26–27
  • Additional metrics should complement compute when scaling laws weaken, but an initial filter must remain based on metrics measurable easily and early in the model lifecycle.Source: §6.4.4 · p. 28

7 Conclusion

Compute thresholds remain a practical tool for identifying potentially risky GPAI models and triggering oversight, but they are not sufficient for regulation. Their use should evolve through research, enforcement design, and combination with complementary metrics.

7 Conclusion

  • Compute thresholds are currently a key tool because they correlate with risk, are measurable and externally verifiable, resist circumvention, and target well-resourced actors.Source: §7 · p. 28
  • Further research should address threshold updates, complementary metrics such as capability evaluations and users, and enforcement across regulatory jurisdictions.Source: §7 · p. 28
  • Compute thresholds should remain one component of a context-dependent regulatory framework, with capability thresholds treated as supplemental rather than competing replacements.Source: §7 · p. 28
Loading 2405.10799v2…