Detecting AI Training in Datacentre Power

1. Introduction

• Off-chip physical sensors such as power meters can be retrofitted to datacentres without trusting operator-controlled hardware or personnel.Other explored channels include on-chip logs, network taps, and personnel-based approaches.Source: §1 · p. 1
• Earlier GPU-level studies report 97% separation with a 10 MHz current probe and 98.2% on-chip telemetry accuracy, with telemetry falling to 43–87% against the hardest disguised workloads (Gargiulo & Kulp, 2026; Rahman & Tajdari, 2026).Facility meters are expected to sample around 10 Hz, far slower than board probes, and sit behind a power-delivery path that can attenuate the signal.Source: §1 · p. 1
• Facility-level detection is harder because real training cadence wanders over seconds to hours, while an operator can deliberately smear the cadence by changing iteration timing or work.The relevant question is whether training can be hidden cheaply, since schedule distortions cost throughput, learning efficiency, or both.Source: §1 · pp. 1–2
• With sampling at ≥2 Hz and integration of ≤0.5 s, Viterbi tracking detects training in almost every five-minute window, within 0.04 of the optimal ceiling.The paper reports that fixed-frequency tests fail once cadence drifts.Source: §1 · p. 2
• 88% of windows are hidden from fixed-frequency tests by work variation at no throughput cost, but defeating Viterbi tracking requires extreme variation and 93–267% more compute for the same loss.Over a day, the strongest tested strategy evades detection in 13% of runs at a 0.1% run-level false-alarm rate.Source: §1 · p. 2
• The conclusion is limited to periodic-schedule detection: training-like non-training loads can also trigger the detector, and the evidence is synthetic rather than measured at a multi-GPU facility.The traces are generated from a model rather than facility measurements.Source: §1 · p. 2

2. A model of datacentre training power

• The power traces are generated synthetically using Ko & Zhu’s stochastic training model, modified with an Ornstein–Uhlenbeck drift in training frequency.Facility-scale measurements are proprietary, so the model serves in place of empirical data.Source: §2 · p. 2

2.1. Training

• Training alternates high-power compute and low-power communication phases; its iteration period varies around a run-specific baseline cadence, with added slow drift to represent persistent slowdowns.The model draws phase ratios and intra-phase fluctuations across iterations and workloads.Source: §2.1 · pp. 2–3

2.2. Fine-tuning

• Fine-tuning uses the training model structure with a lower cadence band, longer tail phase, deeper tail-to-idle swing, and smaller level offsets and intra-phase noise.Cadence drift is applied only to the dominant run; fine-tuning and background training runs have fixed cadence.Source: §2.2 · p. 3

2.3. Inference

• The paper constructs an inference model with broadband decode noise, a slow envelope for expert-load imbalance, and Poisson-arriving prompt-processing pulses; inference has more in-band power but about an order of magnitude less spectral contrast than training.The inference structures follow serving-system behavior, while their amplitudes are chosen to make inference a loud confound.Source: §2.3 · p. 3

2.4. Aggregate

• The aggregate places one dominant training run over smaller training and fine-tuning jobs, while the null replaces the dominant run with inference and keeps the background unchanged.Peak power is divided across the dominant and background workloads; background training contributes weak lines in the search band.Source: §2.4 · pp. 3–4

3. Measurement and threat model

• The hypothesis test compares the declared-workload null with an alternative in which inference is replaced by a dominant training run.The verifier observes a scalar time series from a meter outside the operator’s control.Source: §3 · p. 4, §3.1 · p. 4

3.1. The measurement model

• The observation map filters facility power through delivery and instrument stages, samples it, and adds interference and meter noise; the nominal channel passes power unchanged and samples at 20 Hz.The meter’s integration window attenuates faster variations, while additive terms include noise and possible drifting baselines or controller limit cycles.Source: §3.1 · pp. 4–5, Figure 2 · p. 6
• The channel model assumes a linear time-invariant delivery path and omits amplitude quantisation; claims about off-chip signals remain predictions pending simultaneous device and external-meter measurements.The actual delivery path may depend on operating point.Source: §3.1 · p. 6
• A 1 s integration window cuts cadence signal-to-noise by about two orders of magnitude, and 1 Hz sampling removes the line from the search band; nominal meter noise is not the main in-band limit.At the nominal setting, workload power dominates meter noise in the search band.Source: §3.1 · pp. 5–6

3.2. The threat model

• The verifier owns a tamper-evident meter, while the operator controls workloads, scheduling, other facility loads, and the delivery path; the study tests adversarial strategies individually.The threat model excludes meter tampering, semantic decoys, and training that moves outside the searched band or abandons synchronous iteration.Source: §3.2 · pp. 6–7

4. Detection method

• The detector distinguishes synchronous training from inference by following the training cadence as it wanders rather than testing for a fixed-frequency line.Training produces a strong, narrow spectral line, while inference spreads its power across the band; real-run drift and deliberate schedule reshaping can defeat fixed-frequency tests.Source: §4 · p. 7

4.1. Viterbi line tracking

• A Viterbi tracker scores the best slowly varying path through a frame-normalized spectrogram, penalizing each frequency-bin step between frames.The frame normalization prevents slow changes in overall power from masquerading as a line; dynamic programming finds the best path in time linear in the number of frames.Source: §4.1 · pp. 7–8

4.2. Setting the threshold

• The tracker declares training when its score exceeds an empirical null quantile chosen for false-alarm rate α, typically 10^-2.The null includes weak lines from background training and fine-tuning runs.Source: §4.2 · p. 8

4.3. Baseline methods

• The spectral matched filter tests the band’s peak power against its median, while the multitaper F-test tests for a line at a fixed frequency.A wandering line smears across bins and weakens the matched-filter peak; the F-test requires a line coherent over the whole trace.Source: §4.3 · p. 9

5. Detection results

• On synthetic meter traces, the evaluation tests whether the Viterbi tracker detects the training frequency, comparing H1’s dominant run over 4 small trainings and 4 fine-tunings with H0’s null aggregate and inference in the dominant slot.The default shares are 9 : 0.5 : 0.5, and both classes carry the background’s weak lines.Source: §5 · p. 9

5.1. Detection rate under cadence wander

• At a 10^-2 false-alarm rate, the Viterbi tracker detects 1.00 of runs through 0.4 Hz drift, 0.99 at 0.8 Hz, and 0.96 at 1.5 Hz, staying within 0.04 of the optimal ceiling.The spectral matched filter falls from 0.98 at zero drift to 0.23 at 0.4 Hz; the multitaper F-test has power ≤0.03 across drift levels.Source: Figure 5 · p. 10, §5.1 · p. 10

5.2. A minimum meter specification

• Detection stays near 1.0 at sample rates of 2 Hz or higher, but falls to 0.33 at 1 Hz and chance at 0.5 Hz.The simulated meter has no anti-alias filter; with a sharp anti-alias filter, the sample rate would need to be about 3.4 Hz to retain the full search band.Source: §5.2 · pp. 10–11, Figure 6 · p. 11
• With an honest operator, detection holds at τ ≤0.5 s and without a deep cadence-band notch, alongside the sampling requirement.A 1 s integration window collapses detection to 0.32; a 90–100% notch at 1.0–1.2 Hz reduces it to 0.45–0.63.Source: §5.2 · p. 11

6. Detection against a concealing operator

• The evaluation sweeps eight schedule strategies against an aggregate inference null, measuring detection at a 10^-2 false-alarm rate across 200 traces per class.Throughput costs were measured on a single A100 for four strategies; the other four had no hardware cost measurement.Source: §6 · p. 11
• The Viterbi tracker detects at least 95% of runs in 43 of 46 tested cells; work variation at σG = 0.7 hides 0.55 at approximately zero throughput cost.It detects every run under five strategies and at least 97% under drift; two jitter settings also evade it to a lesser extent.Source: §6 · p. 11
• The work-variation model predicts 53–184% more samples at σG = 0.7 for Bnoise between 8 and 128; long accumulation blocks reduce optimizer steps per token.The strategy includes iterations accumulating 64 or more micro-steps, raising the mean from 8 to 25.Source: §6 · p. 12
• At σG = 0.7, work variation costs 267 ± 33% in training efficiency; lognormal work variation at s = 1.0 hides 0.73 and costs 93 ± 6%.Retuning the optimizer still leaves the lognormal strategy with an 81 ± 5% cost.Source: §6 · p. 12

7.1. Detection probability for long runs

• The Viterbi detector catches at least three quarters of windows for most concealing strategies; work variation is the only tested strategy that hides a large fraction.Figure 7 shows that σG = 0.7 work variation hides 0.55 at approximately zero throughput cost, while idle-insertion jitter hides at most 0.24 at 159% cost.Source: §7.1 · p. 12, Figure 7 · p. 13

7.2. Confounds

• A detection indicates a persistent training-shaped periodic schedule, not necessarily training; ordinary periodic workloads can also trigger the detector.Examples include non-ML kernel loops, periodic inference requests, controller limit cycles, and sustained prefill or batch inference.Source: §7.2 · pp. 13–14

7.1. Detection probability for long runs

• 87% of lognormal work-variation runs are flagged after a day at a run-level false-alarm rate of 10^-3, while the strategy incurs a 93% learning cost.The remaining 13% survive a day; 25 runs have e-values that do not grow on average.Source: §7.1 · p. 13, Figure 8 · p. 14
• Every σG = 0.7 work-variation run is flagged within six hours, including runs detected in only 8–20% of windows.Runs with cadence above about 1.1 Hz have the lowest per-window detection rates, yet are detected at several times the null’s 1% rate.Source: §7.1 · p. 13, Figure 8 · p. 14

7.3. Limitations

• The primary evidence is synthetic: the training model is anchored to published measurements, but facility-scale measurements were unavailable and the inference model lacks validation against measured traces.Single-A100 measurements are a poor proxy for facility-meter detection because they lack the deep compute–communication contrast of synchronized accelerators and measure power through on-card sensors.Source: §7.3 · p. 14
• The false-alarm threshold is guaranteed only when a real facility’s null resembles the simulated null, and day-long guarantees assume independent null-window p-values.A deployed verifier would need calibration against that facility’s own training-free traces or windows.Source: §7.3 · p. 15
• The results cover synchronous, iteration-structured training in the searched cadence band and the tested concealment strategies, not all possible operator strategies.Energy storage could smooth the cadence at the assumed meter location; if storage is inside racks, the meter must move closer to the chips.Source: §7.3 · p. 15

• A passive off-chip power meter can detect synchronous training hidden in a declared inference workload.The Viterbi detector follows the training iteration cadence and is robust to many tested concealing strategies.Source: §8 · p. 15
• Only varying the real work per iteration weakens the detector among the tested strategies, and an anytime-valid test catches most attacked runs within a day.Source: §8 · p. 15
• The strongest tested attack evades detection in about one run in seven after a day, but needs 93% more compute to reach the same loss.Source: §8 · p. 15

A. Optimality ceiling

• The ceiling compares how well each class’s average power spectrum explains a trace’s in-band periodogram, marginalizing the training likelihood over 61 cadence templates.The templates use a uniform prior and are estimated from 2000 simulated traces each; the null spectrum is estimated from null aggregates.Source: A. Optimality ceiling · p. 18
• The ceiling is a benchmark, not a usable detector: it assumes exact class spectra unavailable to a verifier monitoring a real facility.It uses the power spectrum alone, ignoring phase relations between harmonics and non-Gaussian structure.Source: A. Optimality ceiling · p. 18
• The forward-algorithm Neyman–Pearson statistic raises detection from 0.45 to 0.61 at σG = 0.7 and from 0.28 to 0.36 under lognormal work variation.It matches the tracker on honest and drifting runs, so most work-variation hiding persists even under the statistic optimal for the tracker’s model.Source: A. Optimality ceiling · p. 18
• The approximate ceiling detects every trace at every drift, while the tracker’s shortfall is at most 0.04 at σf = 1.5 Hz.This indicates that the reported shortfall is a detector limitation rather than an inseparability of the simulated classes.Source: A. Optimality ceiling · p. 18

B. Fixed-frequency baselines

• The multitaper F-test fits a line and compares its power with nearby taper-observed power, using K = 7 and NW = 4.Its maximum over the search band needs a multiple-frequency correction; the analytic correction did not work.Source: B. Fixed-frequency baselines · p. 18
• A Bonferroni correction produced false-alarm rates about 3.6–6 times nominal, so both fixed tests were thresholded empirically.The excess occurred on every tested null because the F statistic fluctuates on scales finer than a frequency cell.Source: B. Fixed-frequency baselines · pp. 18–19

C. Generator and detector details

• The generator models training, fine-tuning, inference, and their datacentre aggregate, while the meter applies filtering, sampling, and added noise.Training and inference power are synthesized on a 20 Hz grid; the nominal meter noise is white with standard deviation 4 W.Source: C. Generator and detector details · p. 19
• Work variation changes each iteration’s compute-phase micro-step count; its reciprocal form has a heavy tail that raises the mean count from 8 to 25 at σG = 0.7.The lognormal form keeps the mean at G0; other listed strategies alter phase levels, waveform smoothing, background share, or cadence placement.Source: C. Generator and detector details · pp. 19–20
• Tracker frames last 16 seconds at sampling rates of at least 4 Hz and 32 seconds at 2 Hz; window-level rates use the largest threshold passing at most 2 of 200 null traces.At rates at or below 2 Hz, the matched filter uses a single periodogram of the whole trace.Source: C. Generator and detector details · p. 20
• Against the best honest recipe, the best lognormal attacker recipe costs 81 ± 5%; scaling each step’s learning rate by Gi/G0 raises the cost to 116 ± 5%.These measurements use three seeds; the σG = 0.7 attacker was not retuned.Source: C. Generator and detector details · p. 20

D. Meter specification details

• At 1.0–1.2 Hz, deep notches reduce detection to 0.45–0.63; no notch in the sweep removes detection entirely.A 50% depth notch costs nothing at any centre, while a notch at 0.6 Hz costs at most 0.02.Source: D. Meter specification details · p. 20, Figure 9 · p. 20
• Removing the background load changes detection by at most 0.09 and leaves the detection region, fs ≥2 Hz and τ ≤0.5 s, unchanged.The sweep compares a single training run against inference alone with the nominal meter.Source: D. Meter specification details · p. 20

E. Throughput-cost measurements

• Each throughput cost averages eight A100 traces; work-variation results are reported as throughput deficit d, related to c by c = d/(1 − d).The campaign uses a decoder-only transformer trained on random tokens, with fixed gradient-accumulation micro-steps per step.Source: E. Throughput-cost measurements · p. 21
• Jitter and drift insert sleep pads, while work variation changes each step’s micro-step count and never sleeps.Shape filling replaces pad sleep with matrix-multiply bursts; its measured costs include base jitter and are upper bounds.Source: E. Throughput-cost measurements · p. 21
• The cost campaign uses throughput measurements only, because a single card’s measured power line is a poor guide to facility-meter readings.Thus, the reported costs do not depend on that power-measurement caveat.Source: E. Throughput-cost measurements · p. 21

F. Non-training loads that score as training

• All three rules achieve area under the ROC curve 1.00 and retain that score under shifts in headroom, trace length, cadence band, controller lines, or heavy coloured noise.The rules are a fixed standardized-feature score, logistic regression on eight features, and a random forest on waveform-shape features.Source: F. Non-training loads that score as… · p. 21
• At a false-positive rate of 10−2, every rule scores all five non-training control loads as training in at least 97% of traces.The controls include decoy training, gradients without the optimizer swing, a training-shaped non-ML kernel loop, a controller limit cycle, and periodically modulated inference.Source: F. Non-training loads that score as… · p. 22, Table 4 · p. 22
• A run diluted to half the meter’s load is caught by every rule, while asynchronous training with drift and phase slip is caught in 47–100% of traces.The asynchronous case uses σf = 1.2 Hz and σs = 0.15 periods.Source: F. Non-training loads that score as… · p. 22
• The meter observes only the power schedule, so a decoy reproducing training’s compute-communication alternation yields the same trace without training a model.The practical concern is that benign loads such as controller cycles and kernel loops could flag a compliant facility.Source: F. Non-training loads that score as… · p. 22