Source-linked AI summary
LLMs Can Tackle High Energy Theoretical Physics, yet Gaps Remain: Part I, Quantum Field Theory
Tong Liu, Bohan Li, Gongrui Cheng, Zixiong Yu, Yulan He, Volker Tresp
TL;DR
How far LLM reasoning extends into theoretical physics remains unclear, particularly for QFT’s demanding derivations and limited verifiable training examples. The paper introduces QFT-Bench and evaluates 39 models, finding substantial capability among the strongest proprietary systems alongside persistent errors and reasoning unfaithfulness.
Problem
The extent of LLM capability in theoretical physics is unclear, and QFT combines demanding derivations with fewer verifiable training examples than mathematics or code.
Source: §1 · p. 3Method
The authors introduce QFT-Bench, a 75-problem benchmark with closed-form answers, and evaluate models using symbolic checking with an LLM-judge fallback while assessing chain-of-thought faithfulness.
Source: §1 · p. 3, §5.3 · p. 10Results
The strongest proprietary models solve most QFT-Bench problems, while open-source and earlier proprietary models lag; errors persist and reasoning faithfulness increases with capability.
Source: §1 · pp. 3–4, §6 · p. 12Takeaways & Limitations
QFT-Bench provides a first step toward quantitative evaluation of LLM reasoning in QFT and broader high-energy theoretical physics.
Source: §6 · p. 12A correct boxed answer can be unsupported by the displayed calculation, so answer correctness alone does not establish faithful reasoning.
Source: §5.3 · p. 12
Abstract
from arXiv · showhide
Beyond math and code, Large language models (LLMs) are beginning to tackle problems that have traditionally required substantial training for theoretical physicists, but it remains unclear how far this capability extends and where it might break. As a first step toward pushing LLMs in high-energy theoretical physics, we introduce QFT-Bench, a benchmark of 75 graduate- to research-level problems requiring extensive reasoning and derivation across core Quantum Field Theory (QFT) subareas. Each question has a well-defined target and closed-form reference answer. Evaluation combines symbolic verification with LLM-based judge. Across 39 proprietary and open-source models, frontier models already demonstrate substantial capabilities on nontrivial QFT calculations, with the strongest models exceeding 95% accuracy. At the same time, clear gaps remain. Even strong models still make errors in calculations, conventions, and instruction following, while weaker models often fail at the level of physical concepts or derivations. We further monitor models' CoTs and show that current models exhibit nontrivial reasoning unfaithfulness, i.e., they could reach correct answers but the reasoning itself does not justify and logically support them. These results provide a quantitative assessment of current LLM capabilities in QFT and a first step toward broader evaluation in high-energy theoretical physics.
1 Introduction
QFT is a demanding test of LLM reasoning because its derivations are lengthy, depend on physical assumptions, and have fewer verifiable training examples than mathematics or code. The paper introduces QFT-Bench and evaluates current models, finding substantial capability alongside persistent errors and unfaithful reasoning.
1 Introduction
- QFT poses a demanding test because its multi-step calculations depend on physical assumptions and approximations, while its technical, fragmented literature offers fewer verifiable training examples.The paper situates this challenge within broader limits of applying LLMs to theoretical physics, where problems are less formally specified and validity can depend on physical plausibility and experimental evidence.Source: §1 · p. 3
- Across 39 models, the strongest proprietary systems solve a substantial fraction of QFT-Bench, while earlier proprietary and open-source models lag and errors remain.The gap is wider for smaller open-source models.Source: §1 · p. 3
- QFT-Bench comprises 75 graduate- to research-level problems across eight subareas, each with a closed-form reference answer.Correctness is checked symbolically, with an LLM judge used when symbolic verification is inconclusive.Source: §1 · p. 3
- The benchmark evaluates 39 proprietary and open-source reasoning models and classifies incorrect responses into conceptual, calculation, strong-prior, and instruction-following errors.The authors also monitor models’ chains of thought for reasoning faithfulness.Source: §1 · p. 3
- Current models exhibit nontrivial reasoning unfaithfulness, even when their final answers are correct.Faithfulness measures whether a model’s chain-of-thought derivation logically supports its final answer.Source: §1 · pp. 3–4
2 Related Work
Prior work has applied LLMs to theoretical-physics research and developed physics benchmarks, but systematic coverage of QFT reasoning remains limited. QFT-Bench is positioned to provide broad subfield coverage with verifiable answers and reasoning solutions.
LLM Reasoning for Theoretical Physics
- LLMs have been used to derive intermediate steps, conjecture formulas, generate hypotheses, and speed up theoretical-physics calculations, while specialized models have also been fine-tuned or trained from scratch.The authors report that general-purpose frontier models can already solve challenging QFT problems.Source: §2 · p. 4
Existing Theoretical Physics Benchmarks
- Broad expert benchmarks offer limited QFT coverage, undergraduate physics benchmarks focus on earlier topics, and specialized research-level benchmarks cover selected problems.The cited comparison table covers representative benchmarks by HEP scale, difficulty, and physics specialization, but the passage supplies no cell values.Source: §2 · p. 4, Table 1 · p. 4
- Systematic evaluation of QFT reasoning remains relatively limited, motivating broad coverage of core subareas with verifiable answers and reliable chain-of-thought solutions.The benchmark comparison is framed across High-Energy Physics scale, difficulty level, and physics specialization.Source: §2 · p. 4, Table 1 · p. 4
3 QFT-Bench
QFT-Bench covers graduate-level through early research topics across eight QFT categories, with emphasis on supersymmetric and conformal field theory. Its problems are designed to test reasoning, made self-contained, and paired with reference solutions and a two-stage correctness check.
3.1 Overview and statistics
- QFT-Bench contains 75 graduate-level to early research problems grouped into eight QFT topic categories.Around 20% of questions concern each of supersymmetric field theory and conformal field theory, alongside six core areas.Source: §3.1 · p. 4, Table 2 · p. 5
3.2 Dataset Construction
- Problems are designed to test QFT reasoning rather than formula recall, using original tasks and adapted problems rewritten as self-contained questions.Authors identify the central concept, required derivation or calculation, and reference answer when designing original problems.Source: §3.2 · p. 5
- Preliminary model evaluations and manual review identify ambiguities, which are resolved by specifying assumptions, notation, conventions, and target quantities.Reference chain-of-thought solutions are prepared for each problem and checked for consistency with its assumptions and conventions.Source: §3.2 · p. 5
3.3 Verification
- A SymPy-based grader first checks symbolic equivalence to the reference answer, with an LLM judge comparing the solution to the reference when that check is inconclusive or fails.The judge assigns a binary correctness score for mathematical and physical equivalence.Source: §3.3 · p. 5
4 Experimental Setup
The experiments evaluate 39 proprietary and open-source models with thinking enabled when available. Average binary accuracy across problems is the primary metric, and inference uses a common token limit with other parameters left at defaults.
4.1 Language Models and API Setup
- The study selected 39 representative state-of-the-art LLMs, comprising 22 proprietary closed-source models and 17 open-source models, with proprietary examples spanning Claude, GPT, Gemini, and Muse.The open-source examples include the Kimi family.Source: §4.1 · p. 6
- Inference uses a 65,536-token maximum, default settings for other parameters, and thinking mode when available.Claude, GPT, and Gemini are accessed through their official APIs; other models use OpenRouter.Source: §4.1 · p. 6, §4.2 · p. 6
4.2 Metric and Verification
- Average accuracy across all problems is the primary metric, with each problem scored as correct or incorrect.The authors note that future work may extend evaluation to accuracy@k.Source: §4.2 · p. 6
5 Results
The strongest frontier models perform well on QFT-Bench, but errors persist and vary with model performance. Correct final answers can also be supported by reasoning that does not justify them.
5.2 Error analysis
- Conceptual errors dominate among weaker models but become uncommon among stronger ones, whose errors can still involve calculations, familiar formulas, or answering the wrong quantity.In one example, an incorrect vector-leg count leads to an incorrect result.Source: §5.2 · pp. 7, 9
5.1 Overall performance
- Twelve models achieve over 95% accuracy on QFT-Bench; newer models generally outperform predecessors within families, and proprietary models generally outperform open-source ones.Table 4 reports accuracy on QFT-Bench, including symbolic scores and final accuracy.Source: §5.1 · p. 6, Table 4 · p. 8
5.2 Error analysis
- Models may derive the requested quantity correctly yet replace it with a familiar formula or normalization, even when the question specifies a different convention.One described case rescales a Noether charge despite the problem fixing its normalization.Source: §5.2 · pp. 9–10
- Some responses are counted incorrect for failing to provide the requested boxed answer, even when the solution identifies the correct vacua.Some responses also stop before reaching a conclusion.Source: §5.2 · p. 10
5.3 Faithfulness
- Across 39 models, the average unfaithfulness rate is 7.2%; Llama models show roughly 20–40% rates, consistently higher than Qwen models.A response is unfaithful when its final answer is correct but its derivation does not justify it.Source: §5.3 · p. 10
- Higher model capability is associated with greater reasoning faithfulness, with approximately linear improvement within model families.Case studies show correct final results accompanied by incompatible or unsupported intermediate calculations.Source: §5.3 · pp. 10–12
6 Outlook and Conclusion
The conclusion presents QFT-Bench as a first step toward evaluating LLMs in theoretical physics, finding that the strongest proprietary models solve most QFT problems while errors remain. It also outlines evaluation criteria for answer correctness and reasoning faithfulness.
6 Outlook and Conclusion
- The paper asks whether LLMs can contribute to theoretical-physics discovery, noting that breakthroughs have required new ways of understanding nature, not calculations alone.Source: §6 · p. 12
- The strongest proprietary models solve most QFT-Bench problems, but errors remain; open-source models and recently released proprietary models lag behind.The authors describe QFT-Bench as a first step toward evaluating advanced QFT derivations and calculations.Source: §6 · p. 12
- QFT-Bench is a first step toward the broader goal, and the authors plan to extend evaluation to frontier problems in high-energy and theoretical physics.Source: §6 · p. 12
A LLM-as-a-judge Prompt
- The answer-judging prompt checks whether the boxed answer gives the requested quantity and form under the problem’s convention, with missing boxed answers marked wrong.Source: A LLM-as-a-judge Prompt · p. 16
B Reasoning Faithfulness Prompt
- The faithfulness prompt checks whether a boxed answer follows the response’s derivation, separately from whether the answer itself is correct.It treats only inconsistencies that affect the final answer as evidence of unfaithfulness.Source: B Reasoning Faithfulness Prompt · pp. 16–17
- Faithfulness examples include compensating errors or replacing a computed value with a recalled one; harmless typos and corrected slips do not count.Source: B Reasoning Faithfulness Prompt · p. 17
C.1 Category
Incorrect responses are classified by their decisive cause, with an LLM-based judge assigning one category. The accompanying table describes failure-category shares for two model groups.
C.1 Category
- An LLM-based judge assigns each incorrect response one error category based on the decisive cause of its final answer, rather than minor slips.Source: §C.1 · p. 17
- Instruction-following errors include giving no boxed answer or the wrong quantity or form, even when the underlying physics may be correct.Source: §C.1 · p. 17
- Table 5 compares failure-category percentages within the top-half highest-scoring models and the remaining models.Source: Table 5 · p. 18
C.2 Human evaluation
The human evaluation compares three QFT experts’ assessments with two LLM judges on sampled model responses. The paper also presents example problems with reference solutions and correct model responses.
C.2 Human evaluation
- The evaluation sampled 60 responses from 15 models, with four responses per model, and compared three QFT experts’ judgments with two LLM judges.Source: §C.2 · p. 18
- Correctness agreement with human evaluation was 98% for Opus-5-as-a-judge and 92% for GPT-5.6-as-a-judge.Source: §C.2 · p. 18
D Example Problems
- The examples section provides one problem from each of eight categories, along with its reference answer, reference solution, and a correct model response.Source: D Example Problems · p. 19
Classical
The Classical block presents worked QFT calculations spanning fermion scattering, propagator inversion, Bhabha scattering, and one-loop Yukawa renormalization.
Classical
- Identical-fermion scattering includes direct and exchange contractions, with chiral projectors eliminating their interference and initial-spin averaging contributing a factor of 1/4.For the four-fermion ψψ example, the exchange contribution is |M̄2|² = 8λ²u².Source: Classical · pp. 19–20
Quantization
- The massive-vector propagator is obtained by inverting the momentum-space quadratic operator using a tensor ansatz.The solution identifies coefficients multiplying ημν and pμpν.Source: Quantization · p. 21
QED
- Bhabha scattering combines s- and t-channel photon diagrams and has the expected Rutherford-like forward divergence as t → 0 (θ → 0).Source: QED · pp. 22–23
Renormalization
- There is no O(λg) contribution because attaching the ϕ4 vertex to the fermion line requires at least two loops, while the tadpole contraction vanishes.For N = 1, βg = 5g³/16π², and the coupling is not asymptotically free.Source: Renormalization · p. 24
Non-Abelian
The section works through a non-Abelian θ-term, Higgs decay to fermions, and finite-singularity monodromies in pure SU(2) Seiberg-Witten theory.
Non-Abelian
- The non-Abelian θ-term is written locally as a total derivative using the standard topological Chern-Simons current.Source: Non-Abelian · pp. 25–26
Electroweak
- The Higgs-to-fermion tree-level decay-rate derivation proceeds from the Yukawa amplitude through final-state spin and color sums and two-body phase space.Source: Electroweak · pp. 26–27
SUSY
- At the two finite singularities, a monopole and a dyon become massless, and the solution relates their charges and counterclockwise loops to the monodromies.The specified charges are (nm, ne) = (1, 0) at u+ and (1, −1) at u−.Source: SUSY · p. 28
CFT
The CFT section derives OPE expansion coefficients by matching an operator-product expansion to a conformal three-point function.
CFT
- The coefficients are fixed by matching the OPE expansion to the conformal three-point function with normalized two-point functions.Source: CFT · pp. 29–30
3. Matching Coefficients
- Matching the quadratic OPE expansion gives β = (∆ + 2)/[8(∆ + 1)] and γ = −∆/[8(∆ + 1)(2∆ − d + 2)].Source: §3 · p. 31