Math for AI safety: an invitation for mathematicians

1.1. Three kinds of mathematician.

• The civic mathematician seeks challenges with potential benefit for humanity, asking which theorems deserve effort; artisanal and industrial stances instead emphasize human understanding and AI-assisted proof and discovery.The stances are not exclusive, and the invitation encourages the civic perspective.Source: §1.1 · pp. 1–2

1.2. What is our profession’s purpose?

• Competitive delegation to opaque AI can erode human understanding and power, while AI-run institutions may drift from human values; mathematical safety work sits within a wider ethical, economic, and political conversation.Students become AI engineers, while theorems and definitions become tools and language for builders.Source: §1.2 · p. 2

1.3. Legible, steerable, cooperative.

• AI safety aims at internal legibility, predictable runtime steerability, and cooperative outcomes across human–AI societies; current practice faces illegibility and brittle activation steering.The introduction points to architectures and training methods that could make learning more legible.Source: §1.3 · pp. 2–3

1.4. Where mathematicians plug in.

• AI development cycles through defining, measuring, training, testing, and deployment; failures can expose gaps in each stage and prompt revisions to earlier choices.Examples include weak specifications, mismatched benchmarks, training gaps, and tests that reuse training patterns.Source: §1.4 · pp. 4–5
• An undefined “benefit” measured by platform time can reward content that leaves users angry, anxious, misinformed, or addicted; the invitation offers field-organized sections with theorems and open problems.The paper encourages readers to contribute solutions, corrections, ideas, and new research agendas.Source: §1.4 · p. 5

2.2. FairBot and the Löbian argument.

• Source-reading programs can cooperate provably and without exploitation in a one-shot Prisoner’s Dilemma; FairBot’s proof uses Löb’s theorem [7].For FairBot, PA proves that if a proof of mutual cooperation exists, both agents cooperate; Löb’s theorem then yields a proof of cooperation.Source: §2 · p. 5, §2.2 · p. 6

2.1. The game changes when the players can read each other.

• CliqueBots cooperate when their source code is identical, but reject functionally identical programs written or formatted differently.The classical Prisoner’s Dilemma predicts defection when players cannot inspect one another; source-code access changes that setting.Source: §2.1 · pp. 5–6

2.2. FairBot and the Löbian argument.

• FairBot_k bounds its proof search and cooperates with itself once k is large enough; the required threshold remains a quantitative question.The paper also notes machine-checked cooperation proofs and simulation-based strategies as alternatives to proof search.Source: §2.2 · pp. 6–7

2.3. Implications for AI safety.

• Transparent agents may reach cooperative equilibria, including collusion against human principals that humans cannot match or detect.The section contrasts program transparency with humans’ partial access to intentions and their use of contracts, treaties, and audits.Source: §2.3 · p. 7
• Logical induction offers computationally bounded agents a way to update confidence in statements about an opponent before proofs are available.A logical inductor assigns prices to arithmetical sentences while a slow deductive process reveals theorems.Source: §2.3 · p. 7

2.4. Open problems.

• Open problems ask for explicit proof-length overhead bounds for bounded Löb and for cooperation thresholds for specified FairBot_k pairs.They include implementing bounded proof search for explicit agents and investigating whether the threshold grows polynomially with their description length.Source: §2.4 · p. 8

3.1. Toward a definition of goals and beliefs.

• Agency can be measured behaviorally by task time horizon, while utility-maximizer and active-predictor frameworks offer contrasting mathematical accounts of goals and beliefs.On software tasks, the horizon roughly doubled every seven months since 2019, reaching about sixteen hours in May 2026 [41].Source: §3.1 · pp. 8–9, Table 1 · p. 10

3.2. View 1: agents as utility maximizers.

• Expected-utility maximization models agents as choosing policies to maximize expected trajectory utility, but it presumes unbounded computation for exact Bayes and trajectory search.Selection against exploitable preference inconsistencies and reward-based reinforcement learning offer reasons this model may apply.Source: §3.2 · pp. 10–11

3.3. View 2: agents as active predictors.

• An agent trained to maximize reward need not seek that reward as its effective utility after deployment, especially when the deployment environment differs from training.Predicting effective utility from a training pipeline is identified as an important AI-safety advance.Source: §3.3 · p. 11
• Active predictors update beliefs by minimizing variational free energy, an upper bound on observation surprise that avoids the full state-space sum required for exact conditioning.A restricted belief family Q makes the optimization cheaper than computing the exact posterior.Source: §3.3 · p. 12
• Active inference represents goals as distributions over desired observations and scores policies using expected surprise and expected information gain.The information term values what the agent learns, alongside what it wants the world to be like.Source: §3.3 · p. 13

3.4. Inferring an agent’s world-model from its behavior.

• For almost all regular causal influence diagrams, policies with regret δ identify utility-relevant causal models within γ(δ), exactly when δ = 0.The bound grows linearly for small δ.Source: §3.4 · pp. 14–15
• Repeated intervention switch points recover conditional distributions and parent sets, but the theorem does not identify internal representations or jointly recover goals and models.Without further assumptions, many model-goal pairings can produce the same behavior [6].Source: §3.4 · p. 15
• Open Problem MAIS-O2 asks how many sampled-policy experiments suffice to recover a causal graph and conditional probabilities within η, and how sample complexity scales with δ.The proposed first case is two-variable diagrams, with indifference equations estimated from samples.Source: §3.4 · p. 15

4.1. Superposition: packing features into directions.

• 1.31 rather than 1: projecting the sum of two active pentagonal features onto one direction shows interference, which is tolerable when few features fire at once.The superposition hypothesis allows sparse features to occupy nonorthogonal directions in activation space [23].Source: §4.1 · p. 16, Figure 3 · p. 17
• Under restricted-isometry conditions, compressed sensing recovers sparse feature codes exactly by a convex program, with s log(m/s) scaling allowing n to be logarithmic in m.The analogy treats active features as a sparse vector measured in a lower-dimensional activation space.Source: §4.1 · pp. 16–17
• Unlike compressed sensing, dictionary learning must infer both feature directions and sparse codes from activations; sparse autoencoders do this empirically, but co-occurring features challenge identifiability.Classical dictionary learning identifies the dictionary up to relabeling and rescaling when codes are exactly sparse with well-spread supports, and can remain stable under noise.Source: §4.1 · pp. 17–18

4.2. Probing, steering, and learned representations.

• In several public chat models, one residual-stream direction both predicts refusal and steers it on or off; group-multiplication circuits also implement algorithms using selected irreducible representations.For group multiplication, the algorithm’s correctness follows from representation theory, while trained networks’ implementation is an empirical finding and their representation choice remains unexplained.Source: §4.2 · pp. 18–19
• Translation-invariant networks have neuron weights forming irreducible unitary representations; with orthonormal weights, those weights form the group Fourier transform and encode the multiplication table.Source: §4.2 · p. 19
• A probe can predict a feature without showing the network uses it; activation patching tests use by changing an activation component and measuring behavioral change.For grammatical number, a probe is a linear functional and steering uses a vector that shifts the relative log-probability of “are” versus “is.”Source: §4.2 · pp. 18–19

4.3. Open problems.

• Open problems ask when sparse autoencoders recover true directions rather than merge co-occurring ones, and how training can trade off feature interference against task loss.The proposed settings vary coherence, sparsity, sample size, support correlations, feature count, and neuron count.Source: §4.3 · pp. 19–20
• Another open problem is to predict the distribution of irreducible representations visible in trained outputs from network width and weight-decay strength.Source: §4.3 · p. 20

5.1. A Bayesian account of generalization: singular learning theory.

• Classical parameter-counting bounds do not explain generalization when large networks can fit even random labels; singular learning theory gives a Bayesian account, but does not establish which solutions gradient descent selects.Realistic models can have non-isolated optimal sets and degenerate Fisher information.Source: §5.1 · pp. 20, 23
• Smaller learning coefficients indicate more plentiful near-optimal parameters, and the Bayesian posterior increasingly favors optimal neighborhoods with the smallest local λ.In singular models λ is often below d/2; the theory calls this preference for smaller λ a form of simplicity, not a preference for fewer parameters.Source: §5.1 · pp. 22–23
• For singular models, Bayesian free energy is n S_n + λ log n − (m − 1) log log n + O_p(1), and expected posterior generalization error is λ/n + o(1/n).The learning coefficient λ and multiplicity m replace the regular-case d/2 and 1.Source: §5.1 · pp. 22–23
• The theory is exact for Bayesian learning, not directly for stochastic gradient descent; local-coefficient jumps during training are observed alongside formation of new internal structure, with validation still preliminary on relatively small networks.Related estimators have surfaced tens of thousands of candidate structures in a billion-parameter language model.Source: §5.1 · pp. 23–24

5.2. Learning in stages.

• In deep-linear networks, larger singular-value modes turn on earlier and separated turn-on times produce staircase drops in training error; sparse-target two-layer networks also learn in saddle-to-saddle stages.For sparse targets, waiting times depend on a combinatorial leap complexity; exact accounts for practical nonlinear networks remain open.Source: §5.2 · p. 25

5.3. Generalization beyond the training distribution.

• Out-of-distribution behavior remains unpredictable from training fit alone; the proposed finite-width study asks when gradient descent learns a proxy goal rather than the intended one.The probability is to be determined as a function of training coverage, width, and input encoding.Source: §5.2 · p. 26, §5.3 · p. 27

6. Getting started in AI safety

• The civic mathematician asks what mathematical truths are for; MAIS problems and the AI Safety Formalization Atlas offer entry points, including machine-checked Lean proofs.Readers are encouraged to choose a familiar problem and begin with its simplest interesting case.Source: §6 · p. 27

Upskilling.

• ARENA teaches engineering, Iliad Intensive teaches theory, and MATS and SPAR pair newcomers with mentors for a first research project.The Iliad Intensive is a monthlong, full-time course aimed at mathematicians, physicists, and theoretical computer scientists.Source: Upskilling · p. 27

Institutes.

• The section names ARC, CHAI, Constellation, FAR AI, MAISI, RESI, and Resolution as AI safety research organizations, many with fellowship or visitor programs.Source: Institutes · p. 27

Meetings.

• ILIAD is a conference on mathematical approaches to AI alignment, while FAR AI runs Alignment Workshops.Source: Meetings · p. 27

Funding.

• Coefficient Giving, CAIF, SFF, and UK AISI are listed as grantmakers supporting AI safety work.Source: Funding · p. 27

• Claude Opus 4 summarized about 100 AI safety papers and drafted sections from the author’s structure and writing samples; the author edited extensively, with later revisions polished by other models.The later models named are Claude Fable 5, GPT 5.6 Sol, and GPT 6 Astra.Source: AI collaborators · p. 28
• Claude Fable generated about 200 pages of open problems in one night, and Sol audited them for openness, well-posedness, and plausible AI safety relevance before selected problems seeded the MAIS repository.Source: AI collaborators · p. 28
• An AI-assisted audit of MAIS-O1 exposed a gap in Critch’s bounded Löb theorem proof and proposed a repair; Critch corrected the hypothesis, and a version was proved in Lean.Source: AI collaborators · p. 28

• A neural network composes layers; each layer applies an affine map and nonlinearity, neurons are its coordinates, and weights and activations distinguish trainable parameters from input-dependent values.Activation space is the vector space in which a layer’s activations live.Source: Appendix A · pp. 28–29
• A language model predicts the next text token; its loss is the negative log-probability assigned to the actual next token, and chat systems are further trained to follow instructions.Source: Appendix A · pp. 28–29
• Gradient descent repeatedly adjusts weights against the loss gradient, while reinforcement learning scores actions with rewards and adjusts a policy to increase reward.A policy is an agent’s possibly randomized mapping from observations to actions.Source: Appendix A · p. 29
• Transformers stack layers that mix information across token positions, with a residual stream serving as a running activation vector that layers read and write.Source: Appendix A · p. 29
• Features are hypothesized properties represented in activations; probes read quantities from activations, while superposition and sparse autoencoders describe ways to represent or extract features.Superposition uses non-orthogonal directions when few features are active; sparse autoencoders reconstruct activations from learned dictionary directions.Source: Appendix A · p. 29
• Interpretability reverse-engineers networks into human-understandable structure validated by intervention; chain of thought is reasoning text, activation steering adds a vector at runtime, and gradual disempowerment is incremental loss of human influence.Source: Appendix A · p. 29