Source-linked AI summary
MathBode: Measuring the Stability of LLM Reasoning using Frequency Response
Charles L. Wang
TL;DR
Final-answer accuracy on mathematical benchmarks does not show how models reason or whether their behavior stays stable under controlled changes. MathBode sinusoidally drives a problem parameter and measures first-harmonic gain and phase across five closed-form families. The diagnostic finds low-pass behavior and growing phase lag that static accuracy can obscure.
Problem
Final-answer accuracy and one-input/one-answer math benchmarks do not measure how models reason or whether behavior is stable under controlled changes.
Source: §1 · p. 1Method
MathBode sinusoidally varies one problem parameter and fits first-harmonic model and exact-answer responses to produce frequency-resolved gain and phase fingerprints.
Source: §1 · pp. 1–2Results
Across five closed-form families, models exhibit low-pass behavior and growing phase lag, while the symbolic baseline calibrates the instrument near G≈1 and ϕ≈0.
Source: §1 · p. 1, §4 · pp. 3–4, §5 · p. 5Takeaways & Limitations
Frequency fingerprints provide a compact diagnostic for model selection and ablation studies that complements standard benchmarks with measurements of reasoning fidelity and consistency.
Source: §2 · p. 3, §5 · p. 5At ω=16 the drive approaches the Nyquist limit and may introduce aliasing artefacts, so rankings emphasize the mid-band {4, 8}.
Source: §2 · p. 2
Abstract
from arXiv · showhide
This paper presents MathBode, a dynamic diagnostic for mathematical reasoning in large language models (LLMs). Instead of one-shot accuracy, MathBode treats each parametric problem as a system: we drive a single parameter sinusoidally and fit first-harmonic responses of model outputs and exact solutions. This yields interpretable, frequency-resolved metrics -- gain (amplitude tracking) and phase (lag) -- that form Bode-style fingerprints. Across five closed-form families (linear solve, ratio/saturation, compound interest, 2x2 linear systems, similar triangles), the diagnostic surfaces systematic low-pass behavior and growing phase lag that accuracy alone obscures. We compare several models against a symbolic baseline that calibrates the instrument ($G \approx 1$, $\phi \approx 0$). Results separate frontier from mid-tier models on dynamics, providing a compact, reproducible protocol that complements standard benchmarks with actionable measurements of reasoning fidelity and consistency. We open-source the dataset and code to enable further research and adoption.
1 Introduction
Static final-answer accuracy can obscure how mathematical models reason and whether their behavior remains stable under controlled changes. MathBode probes parameterized problems across frequency to measure amplitude tracking and phase lag.
1 Introduction
- MathBode drives one problem parameter sinusoidally and fits first-harmonic responses of model outputs and exact solutions to measure gain and phase over frequency.The protocol uses deterministic decoding and sweeps frequencies to expose amplitude fidelity, timing lag, and prompt-surface sensitivity.Source: §1 · pp. 1–2
- MathBode applies the diagnostic to five closed-form families, producing Bode-style fingerprints that reveal low-pass behavior and growing phase lag even when static accuracy ties.A symbolic baseline calibrates the instrument at ideal gain near 1 and phase near 0.Source: §1 · p. 1
- Static math benchmarks retain one-input/one-answer evaluation, while robustness and repeated-trial work probes answer flips and inconsistency [Patel et al., 2021, Huang et al., 2025, Wang et al., 2022, Zeng et al., 2023].The paper motivates metrics for reliability and invariance, not just correctness.Source: §1 · p. 1
- Interpretability work on arithmetic transformers’ sinusoidal and rotational codes motivates a frequency-domain probe as more than a metaphor [Nanda et al., 2023, Kantamneni and Tegmark, 2025, Li et al., 2024].The paper frames this connection as motivation for a descriptive lens, not a modeling assumption.Source: §1 · p. 2
2 Benchmark
The benchmark probes model responses to controlled parameter trajectories, fitting first-harmonic behavior against exact solutions. Its frequency grid and diagnostics characterize tracking while bounding interpretation near the highest frequency.
2 Benchmark
- The benchmark treats each parametric problem as a system, driving one parameter sinusoidally and fitting model and exact-answer series to first-harmonic responses.The fitted series provide gain and phase; the symbolic solver runs through the same pipeline as an ideal reference.Source: §2 · p. 2
- The sinusoid is a controlled probe, not an assumption of linear time-invariant behavior; residual diagnostics and H2/H1 capture departures such as nonlinearity and memory.The frequency lens is motivated descriptively by findings on sinusoidal and rotational number codes.Source: §2 · p. 2
- Five fixed-domain closed-form families, each with three question variants, cover linear solve, ratio saturation, exponential interest, linear systems, and similar triangles.Inputs are clipped in-range, and the family domains and trajectory parameters are exposed via code.Source: §2 · p. 2
- The sweep uses T=64, frequencies {1, 2, 4, 8, 16}, three start phases, and perturbations roughly 10% of each family’s half-range.Deterministic temperature-0 decoding and strict numeric parsing with compliance filtering are used.Source: §2 · p. 2
- At ω=16 the drive approaches the Nyquist limit, so possible aliasing in R2 dips or phase swings motivates ranking models in the mid-band {4, 8}.All experiments use temperature 0 and strict numeric parsing with compliance filtering.Source: §2 · p. 2
- Gain and phase summarize amplitude tracking and lag, while R2 and residual diagnostics test the first-harmonic fit and expose unexplained structure.The pipeline also reports residual RMS, residual ACF(1), and a nonlinearity proxy H2/H1; a symbolic solver calibrates the end-to-end instrument.Source: §2 · pp. 2–3
3 Dataset Details
MathBode’s dataset spans five problem families and records both rendered prompts and the parameters and answers needed to analyze each instance.
3 Dataset Details
- The dataset contains 9,408 rows per family and 47,040 rows total across five families.Table 1 presents dataset rows by family.Source: §3 · p. 3, Table 1 · p. 3
- Each dataset record includes family, question variant, signal type, amplitude scale, frequency, phase, time step, parameter value, rendered prompt, and exact answer.The schema identifies the signal as sinusoid, chirp, or step and stores the ground-truth numerical answer.Source: §3 · p. 3
- The dataset schema identifies five families: exponential interest, linear solve, linear system, ratio saturation, and similar triangles.The family names are represented as string-valued entries.Source: §3 · p. 3
4 Evaluation
MathBode scores summarize mid-band gain and phase deviations, with fit and residual diagnostics qualifying first-harmonic responses. Results show frequency-dependent behavior that static accuracy can obscure.
4 Evaluation
- Gain declines with frequency in Linear Solve and Exponential Interest; Similar Triangles stays near G≈1, while Linear System amplifies between-model differences.Figure 1 compares gain across problem families and models against the unity reference.Source: §4 · p. 3, Figure 1 · p. 4
- Phase lag typically grows with frequency; proportional families remain near 0°, while Linear System shows the largest swings.Figure 2 displays signed model–truth phase error across frequencies.Source: §4 · p. 4, Figure 2 · p. 4
- Residual ACF(1) trends toward zero or negative with frequency, indicating little remaining temporal structure or alternating errors rather than drift.Figure 3 shows residual autocorrelation by frequency.Source: §4 · p. 4, Figure 3 · p. 4
- R² is near 1 for Similar Triangles and in the mid-band elsewhere; drops in Exponential Interest and Linear System coincide with the largest gain and phase deviations.These drops point to emergent nonlinearities rather than random noise; Figure 4 plots first-harmonic fit quality.Source: §4 · p. 5, Figure 4 · p. 5
- DeepSeek V3.1 leads overall on both MB-Core and MB-Plus.MB-Plus additionally downweights poor fit quality, residual structure, and nonlinearity.Source: Table 2 · p. 5
5 Conclusion.
The conclusion presents MathBode as a dynamic frequency-domain diagnostic that complements final-answer evaluation. Across five closed-form families, it reports low-pass behavior and growing phase lag, while noting limits in family coverage and probe design.
5 Conclusion.
- Across five closed-form families, models show low-pass behavior and growing phase lag, while static accuracy can mask amplitude and timing errors.The symbolic baseline and MB-Core/MB-Plus scores summarize these dynamics.Source: §5 · p. 5
- MathBode provides interpretable gain/phase fingerprints as reproducible diagnostics that complement standard benchmarks and support model selection and ablation studies.The dataset and reference code are released to support replication and extension.Source: §5 · p. 5
- The evaluation is limited to a small number of families and single-tone drives; future work will expand tasks and add richer inputs.The conclusion also proposes linking frequency fingerprints to internal mechanisms.Source: §5 · p. 5
B Presets
The inference presets specify whether phase repeats are used; MVP_PLUS applies tri-phase repeats only at mid-band frequencies.
B Presets
- Tri-phase uses phases {0,120,240}; MVP_PLUS applies them only at frequencies {4, 8}, while other frequencies use phase {0}.Table 4 identifies whether presets use tri-phase repeats.Source: B Presets · p. 7, Table 4 · p. 7
C Answer Format & Strict Parsing
Responses use a fixed-precision decimal format, and a strict parser extracts numeric answers and classifies rows as compliant or non-compliant.
C Answer Format & Strict Parsing
- The required answer payload is a decimal with exactly six places, enclosed by [answer_start] and [answer_end].The parser truncates to six decimals, padding shorter values with zeros.Source: C Answer Format & Strict Parsing · p. 7
- The parser takes the last complete tag pair and last ASCII decimal literal, rejecting scientific notation, separators, units, non-finite values, and missing tags.Values with more than six decimals are cut off; those with fewer are padded.Source: C Answer Format & Strict Parsing · p. 7
- Only compliant rows enter harmonic fitting and residual diagnostics, while non-compliant rows still count toward compliance statistics.Source: C Answer Format & Strict Parsing · p. 7
D Figures & Tables
The reported diagnostics identify family-specific amplitude and timing errors, near-perfect compliance, and greater nonlinearity and residual structure in EI and LS than in simpler families.
D Figures & Tables
- EI and LS dominate amplitude error; DeepSeek is best on EI gain, while Mixtral is worst on RS.The table reports mean |G−1| at mid-frequencies, with lower values better.Source: Table 5 · p. 7, D Figures & Tables · p. 7
- LS is the timing bottleneck; Qwen is best on LS phase, while Mixtral collapses on RS.The table reports mean absolute phase error in degrees at mid-frequencies, with lower values better.Source: Table 6 · p. 8, D Figures & Tables · p. 8
- Compliance is perfect overall, so the reported dynamics reflect model behavior rather than parse failures.The authors say MB-Plus penalties therefore primarily capture quality, not I/O brittleness.Source: Figure 5 · p. 8, D Figures & Tables · p. 8
- H2/H1 nonlinearity concentrates in EI and LS, while Similar Triangles stays near zero.The authors interpret EI/LS peaks as waveform deformation under parameter sweeps and recommend multi-tone tests or chirps to distinguish memory effects from static nonlinearity.Source: Figure 6 · p. 8, D Figures & Tables · p. 8
- Single-sinusoid residuals are largest in EI and LS, while simpler families fit tightly.High residuals indicate that a first-harmonic model leaves structure unexplained; low residuals support mid-band summaries as compact proxies for simpler families.Source: D Figures & Tables · pp. 8–9, Figure 7 · p. 9
E API Settings
Model calls used fixed decoding settings and rate limits, held constant across experiments unless noted otherwise.
E API Settings
- All model calls used temperature 0.0 and a maximum of 1028 tokens.These were fixed decoding settings for Together and OpenAI calls.Source: E API Settings · p. 9
- Rate limits were 600 requests per minute for Together and 20,000 tokens per minute for OpenAI.The settings were held constant across experiments unless explicitly noted elsewhere.Source: E API Settings · p. 9