Small models, big threats: Characterizing safety challenges from low-compute AI models

• Current AI governance frameworks commonly monitor or regulate compute, including high-performance hardware access and training-FLOP thresholds.Source: §1 · p. 1
• Low-compute models are gaining capabilities through test-time compute, parameter quantization, agentic workflows, and other techniques that diffuse capabilities from larger systems.Source: §1 · p. 1
• The report argues that compressed, accessible models pose significant security risks that existing governance frameworks are not fully equipped to address.Source: §1 · p. 1
• The report profiles open-source LLM performance over time and examines how capability compression changes the compute needed for societal-harm campaigns.Source: §1 · pp. 1–2
• The report defines low-compute AI models as systems with ¡30B parameters and uses model compression to describe capability diffusion into such systems.Source: §1 · p. 2

2.1. Model miniaturization

• The analysis uses over 5,000 HuggingFace open-source LLMs and defines aggregate performance α as the mean score across six benchmark tests.Source: §2.1 · p. 2
• Model-size curves estimate the 25th-percentile size exceeding α values from 30% to 50%, with exponential fits and uncertainty bands.Source: §2.1 · p. 2
• The model size needed to obtain α = 0.35 fell by ∼10X over the past year, while Llama models of all size ranges increased in performance toward ∼45%.Source: §2.1 · p. 2, Figure 1 · p. 3
• Benchmark performance may overstate practical usefulness because models can be engineered or overfit to benchmark tests.Source: §2.1 · p. 2

2.2. Advances in hardware

• Processing power and memory bandwidth increased over time for both NVIDIA data-center GPUs and consumer MacBook GPUs.Source: §2.2 · p. 3
• Unrestricted MacBook chips are sufficient to run many advanced and potentially dangerous models, especially alongside rapid model miniaturization.Source: §2.2 · p. 3, §2.1 · p. 3

3. The threats posed by low-compute AI systems

• The assessment combines public incident trends, studies of low-compute model capabilities, and compute simulations to characterize threats from multiple vantage points.The report treats each vantage point as partial but argues their combination provides a more complete picture.Source: §3 · p. 3

3.1. The rapid rise of AI security incidents

• Reported incidents included $2.9B lost to business-email-compromise scams in 2023, a 1,265% phishing increase, a global sextortion crisis, and AI voice scams affecting 25% of surveyed U.S. adults or their acquaintances.Source: §3.1 · p. 3

3.2. Identification of dangerous capabilities within low-compute AI systems

• Human studies found compressed models produced politically persuasive, spear-phishing, and behavior-shaping outputs, while humans detected StyleGAN2 deepfakes and synthetic audio in only about 60% of instances.Source: §3.2 · p. 4

3.3. Simulating compute budgets of social harm campaigns

• Campaigns are decomposed from historical disinformation, cybersecurity, voice-cloning, and deepfake cases into constituent generation tasks and associated timescales.Historical case studies anchor simulated campaign scales to events with understood social impact.Source: §3.3 · p. 4

3.3.2. THE PERIODIC TABLE OF SYNTHETIC MEDIA ATTACKS

• Compute profiles are measured by profiling image, text, and audio generation on a V100, aggregating constituent-task requirements, and sampling parameter uncertainty with Monte Carlo simulations.The profiler is used instead of theoretical derivation because batch sizes, image dimensions, and other workload details complicate analytic estimates.Source: §3.3.2 · pp. 4–5, §3.3.3 · p. 5

3.3.3. ESTIMATING UNCERTAINTIES

• Most studied societal-harm campaigns fit within the performance bounds of non-export-controlled V100 and M2 Ultra accelerators, while ten V100 chips exceed all estimated campaign upper bounds.The simulations measure memory bandwidth and processing power across campaign tasks; conservative assumptions may overstate required resources.Source: §3.3.3 · pp. 5–6, Figure 3 · p. 5
• The results are limited by assuming generative models below 30B parameters, whose ability to execute every studied campaign is not yet fully established.The report expects required compute to decrease if these models become more capable and efficient.Source: §3.3.3 · p. 6

4. Can’t compute thresholds simply be adjusted to address threats from low-compute AI?

• Restricting compute enough to block harmful campaigns could harm hardware manufacturers, strain international relationships, and hinder research collaborations.The report frames threshold design as a trade-off between reducing AI risks and supporting non-nefarious development.Source: §4 · p. 6
• The analysis profiles legitimate workloads with a procedure similar to the societal-harm simulations and compares their compute requirements directly.The comparison is intended to test whether compute metering frameworks can be revised without catching ordinary workloads.Source: §4 · p. 6
• The workload comparison covers diverse commercial and academic applications selected across model types and sectors, including audio, video, and text generation.Recommendation-engine training was included because it was estimated to be a high-volume data-center workload as of 2019.Source: §4 · p. 6
• Every profiled business or academic workload requires more compute than all studied societal-harm campaigns, so threshold changes targeting those attacks could disrupt legitimate use cases.The selected workloads span autonomous-vehicle recognition, protein prediction, transcription, spam detection, and recommendation-engine training.Source: §4 · p. 6

• Capability-based frameworks evaluate harmful abilities directly, such as persuasion, deception, and dual-use proficiency, then prioritize flagged systems for review or access restrictions.Source: §5 · p. 6
• Capability evaluations require substantial expertise, can become outdated, and may be evaded by developers optimizing systems for benchmarks rather than deployment contexts.Source: §5 · p. 6
• Defensive AI could counter voice-cloning and cybersecurity attacks, but may be less effective against threats where offensive capabilities are easier to deploy than defensive responses.Source: §5 · p. 7
• Inference-time filters and watermarks can block harmful outputs or help people identify synthetic interactions involving cloned voices, LLMs, or generative agents.Source: §5 · p. 7
• Social resilience efforts include media literacy, incident reporting, AI education, and safeguards around datasets or materials needed to execute attacks.Source: §5 · p. 7
• Resilience infrastructure requires funding and cross-sector coordination, while toxicity filters may fail to distinguish socially dangerous content from benign content and models can still be elicited into harmful outputs.Source: §5 · p. 7

• Compute-based policies can limit attacks whose severity scales with compute, but they do not reliably constrain campaigns whose impact arises from modest-scale activity or low-compute catastrophic actions.Source: §6 · pp. 7–8
• A simulated misinformation campaign matched the scale of a documented Brexit campaign of approximately 50,000–100,000 posts, while pathogen design could require minimal compute yet cause catastrophic harm.Source: §6 · pp. 7–8
• The considered low-compute models fall below the EU AI Act systemic-risk compute threshold and can run on non-export-controlled hardware, leaving vulnerabilities in compute-metering policies.Source: §6 · p. 8
• Market incentives may not produce adequate safeguards because organizations report lacking resources, expertise, or incentives, motivating strategic public-private collaboration.Source: §6 · p. 8

• Model sizes needed for competitive LLM benchmark scores decreased by as much as 10X over the past year across historical data from more than 5,000 HuggingFace models.Source: §7 · p. 8
• Many simulated social-harm attacks are easily executable on consumer hardware, showing that increasingly capable low-resource models can support harmful campaigns without frontier-scale compute.Source: §7 · p. 8
• The paper argues that current governance frameworks have not adequately evolved to address increasingly powerful low-compute models and rely too heavily on compute thresholds as risk proxies.Source: §7 · p. 8
• It recommends frameworks that consider capabilities, intent, and potential harm, supported by collaboration among technical experts, policymakers, and industry.Source: §7 · p. 8

• The HuggingFace leaderboard repository provides model sizes and benchmark scores but lacks all information needed to construct the report’s time-based analysis.Source: A. HuggingFace Leaderboard data… · p. 12
• Because evaluation dates may differ from model creation dates, a model trained earlier could appear associated with a later leaderboard evaluation.Source: A. HuggingFace Leaderboard data… · p. 12
• Creation dates are inferred from the first commit uploading a .config.json file, an assumption checked against public announcements for Llama, Mistral, and Phi-2.Source: A. HuggingFace Leaderboard data… · p. 12
• Interim repository revisions can cause the method to associate later capabilities with an earlier creation date, although the authors speculate such errors are likely minimal and do not quantify them further.Source: A. HuggingFace Leaderboard data… · p. 12

• Profiler measurements of matrix-multiplication FLOPs agree well with established theoretical estimates, supporting the profiling approach.Source: B. Compute profiling · p. 12, Figure 5 · p. 13
• The profiled models comprise StabilityAI diffusion, Meta MMS-TTS, OpenAI Whisper, Microsoft Phi-2, and Mistral-7B systems selected for developer-community popularity.Source: B. Compute profiling · p. 12
• Text-model compute was measured across batch size and generated-token count, while image and audio profiles varied dimensions, denoising steps, audio length, and converted tokens.Source: B. Compute profiling · p. 12

• The business-email-compromise simulation assumes attackers already possess corporate email access and assigns the LLM responsibility for orchestrating employee communications.Source: C. Description of case studies · p. 13
• The DeepNude case estimates resources for generating approximately 100,000 synthetic images over one month using open-source diffusion models.Source: C. Description of case studies · p. 13
• The voice-cloning simulation models real-time speech-to-text, LLM response generation, and text-to-speech without profiling optional audio-model fine-tuning.Source: C. Description of case studies · pp. 13, 15
• The Brexit botnet case models 13,000 accounts posting roughly 65,000 divisive tweets over two weeks, while the phishing case targets about 1,000 employees with ten emails each.Source: C. Description of case studies · pp. 13, 15

• The compute formula aggregates image, text-to-speech, speech-to-text, and language-model workloads into processing-power or memory-bandwidth estimates for each campaign.Source: D. Compute profiling of case studies · p. 15
• Monte Carlo trials sample uncertain timelines, generation counts, dimensions, model sizes, token counts, and inference settings while holding sampled values consistent within each trial.Source: D. Compute profiling of case studies · p. 15, Table 1 · p. 17
• Campaign distributions are summarized with 2.5th and 97.5th percentile values, illustrated for the botnet scenario in Figure 7.Source: D. Compute profiling of case studies · p. 15, Figure 7 · p. 16
• Upper bounds receive a tenfold addition for omitted computational tasks, whereas lower bounds are reduced tenfold to account for efficiency optimizations.Source: D. Compute profiling of case studies · pp. 15–16

• Workload estimates use 10–90% GPU performance limits, and the autonomous-vehicle profile excludes trajectory planning and 3D reconstruction as an extremely conservative lower bound.Source: E. Simulation of non-nefarious workloads · p. 17
• The evaluated workloads include AlphaFold inference, call-center transcription, spam detection, and a deep-learning recommendation system.Source: E. Simulation of non-nefarious workloads · p. 17
• The call-center profile assumes 572,000 seconds of transcribed audio, while spam detection processes 800,000 tokens with a roughly 500M-parameter model.Source: E. Simulation of non-nefarious workloads · p. 17
• The recommendation-system estimate scales measured FLOPs from one training mini-batch to a full 300-epoch, 1,000-mini-batch training run.Source: E. Simulation of non-nefarious workloads · pp. 17–18
• The extracted compute profiles for the considered workloads are presented in Table 2.Source: E. Simulation of non-nefarious workloads · p. 18, Table 2 · p. 18