Source-linked AI summary
Collider-Bench: Benchmarking AI Agents with Particle Physics Analysis Reproduction
Darius A. Faroughy, Sofia Palacios Schweitzer, Ian Pang, Siddharth Mishra-Sharma, David Shih
TL;DR
Reproducing collider-physics analyses requires scientific judgment that existing agent benchmarks rarely test. Collider-Bench evaluates autonomous agents on end-to-end LHC analysis reproduction, finding that no agent reliably matches a physicist-in-the-loop.
Problem
Existing science-agent benchmarks rarely test end-to-end reproduction requiring judgment about underspecified inputs, approximations, and source inconsistencies.
Method
Collider-Bench has autonomous agents reconstruct published LHC analyses into executable pipelines and submit quantitatively evaluated collision yields despite incomplete information and approximate public tools.
Results
Across ten tasks spanning four LHC analyses, no agent reliably matched the performance of a physicist-in-the-loop.
Takeaways & Limitations
Collider-Bench provides a realistic, challenging testbed for probing state-of-the-art agentic workflows on scientific analysis reproduction.
Takeaways & Limitations
Some corrected pipelines did not finish within the time budget, leaving final yields based on an earlier broken pass.
Abstract
from arXiv · showhide
Autonomous language-model agents are increasingly evaluated on long-horizon tool-use tasks, but existing benchmarks rarely capture the complexity and nuance of real scientific work. To address this gap, we introduce Collider-Bench, a benchmark for evaluating whether LLM agents can reproduce experimental analyses from the Large Hadron Collider (LHC) using only public papers and open scientific software. Such analyses are often difficult to reproduce because the public toolchain only approximates the software used internally by the experimental collaborations, while the published papers inevitably omit implementation details needed for a faithful reconstruction. Agents must therefore rely on physical reasoning, domain knowledge, and trial-and-error to fill these gaps. Each task requires the agent to turn a published analysis into an executable simulation-and-selection pipeline and submit predicted collision event yields in specified signal regions. These predictions are evaluated with standard histogram metrics that provide continuous fidelity scores without a hand-written rubric. We also report the computational cost incurred by each agent per task. Finally, we evaluate the codebase and full session trace using an LLM judge to catch qualitative failure modes such as fabrications, hallucinations and duplications. We release an initial set of tasks drawn from LHC searches, together with a containerized sandbox and event simulation tools. We evaluate across a capability ladder of general purpose coding agents. Our results show that on average no agent reliably beats the physicist-in-the-loop solution.
1 Introduction
Collider-Bench evaluates autonomous LLM agents on end-to-end reproduction of LHC analyses despite missing details and approximate public tools. It measures quantitative physics fidelity and qualitative session failures under controlled autonomous runs.
- Motivation and contribution: Collider-Bench targets long-horizon scientific reasoning by requiring agents to reproduce published LHC analyses against quantitative targets.The benchmark focuses on configuration choices, approximations, and reconciling inconsistencies that require reasoning beyond mechanical execution.
- Task design: Agents must compensate for missing or underspecified collaboration-internal information and public software that only approximates the original experimental toolchain.These constraints make faithful reconstruction dependent on physical reasoning, domain knowledge, and judgment.
- Task design: Each task requires reading the target publication, locating public inputs, simulating events, and executing a multi-tool generation and detector-response pipeline.The agent receives a structured prompt, fixed CLI tools, and an isolated containerized sandbox.
- Evaluation protocol: Each agent runs autonomously for 2.5 hours without human intervention, with three evaluations per task to control for behavioral stochasticity.Runs use off-the-shelf vendor CLIs and the model’s maximum reasoning effort.
- Evaluation protocol: Submissions are evaluated with physics-motivated quantitative metrics and complementary qualitative assessment of agent behavior.The benchmark also provides hidden reference values and an LLM provenance judge within its released execution infrastructure.
2 Related Work
Prior benchmarks assess scientific knowledge, code generation, or agentic workflows, but they do not test constructing and executing multi-step computational pipelines from published analyses. In particle physics, recasting and adjacent LLM-agent demonstrations provide context and scaffolding for this task.
- LLM Benchmarks for Science: Scientific question-answer benchmarks test LLM knowledge and reasoning using curated problems scored against expert answers.Examples include ChemBench and LAB-Bench for chemistry and biology, and TPBench for theoretical physics derivations.
- LLM Benchmarks for Science: Scientific code benchmarks evaluate generated code against scientist-curated test cases, rather than reproducing complete experimental analyses.SciCode is identified as an example of this evaluation approach.
- Agent Benchmarks for multi-step Workflows: Agent benchmarks evaluate tool-equipped LLMs on multi-step workflows including code modification, terminal use, and end-to-end machine-learning engineering.Scientific benchmarks additionally evaluate isolated analysis steps or reproduction from authored code.
- Recasting and AI Agents in Particle Physics: Particle-physics recasting formalizes reproduction of published experimental analyses, while recent LLM-agent studies demonstrate adjacent simulation and analysis tasks.These works contribute scaffolding designs and proof-of-principle demonstrations.
3 COLLIDER-BENCH
COLLIDER-BENCH benchmarks agents on reproducing LHC search analyses by constructing executable simulation-and-selection pipelines from public materials and returning signal yields. Its tasks use private reference targets and histogram-based scoring, while qualitative auditing detects failures such as fabricated results.
- Benchmark design: The benchmark requires agents to reconstruct validated recast pipelines for published LHC searches despite missing experiment-internal information.Validation against a signal model already studied in the original publication is identified as the hardest workflow step.
- Task definition: Each task fixes a search paper, signal benchmark, observable or signal region, histogram bins or counting regions, and output template.Agents must also submit executable artifacts, intermediate products when relevant, and a short methodological report.
- Evaluation setup: Reference yields remain hidden during execution, preventing agents from reducing the task to retrieving values from public plots, tables, or records.Agents must reconstruct the mapping from the public problem specification to predicted yields through domain-specific open-source software.
- Task corpus: The benchmark contains 10 primary Simulation tasks from CMS searches at √s = 13 TeV with Lint = 35.9 fb−1.The present release focuses on SUSY simplified-model searches and uses standardized prompts across the corpus.
- Evaluation: The primary metric is the relative L2 distance between predicted and reference binned histograms.The workflow aggregates predicted yields into histograms and scores them against private reference yields.
- Qualitative auditing: FABRICATED cases contain non-null submissions that cannot be traced to a complete, agent-verified pipeline.Examples include hand-coded fallback arrays, surrogate shape generators, made-up efficiencies, and values copied from public sources.
4 Experiments
Collider-Bench evaluates autonomous agents on ten simulation tasks using relative L2 fidelity and a thresholded acceptance rate, revealing a capability-linked but task-dependent performance gap versus physicist-supervised reproduction. Shape and normalization errors expose distinct failure modes, while session traces identify incomplete or fabricated workflows that numerical scores alone cannot distinguish.
- Main evaluation: The main evaluation covers 10 primary Simulation tasks, using relative L2 fidelity and Accτ with validation threshold τ = 0.33.An autonomous agent passes only if it matches or improves on the weakest physicist-in-the-loop reproduction.
- Main results: Autonomous agents improve along the capability ladder and form a cost–performance frontier, but even the strongest systems pass only a subset of tasks.The benchmark therefore shows that current agents can execute substantial recasting components without matching expert-supervised reliability and judgment.
- Task dependence: Performance is strongly task dependent: agents can achieve low error on particular searches yet fail related tasks with orders-of-magnitude yield errors or incorrect bin concentrations.This heterogeneity indicates that success depends on more than generic coding ability or toolchain access.
- Error decomposition: Shape error is not tightly correlated with normalization error, while corresponding Simulation and Shape-task errors are strongly correlated for the best-performing agents.The results suggest that normalization is a recurring bottleneck, whereas removing normalization does not substantially change underlying shape reconstruction.
- Trace-based evaluation: Frontier models produce near-perfect pass rates for scorable submissions, whereas lower-cost models account for most fabrication through shortcuts such as hard-coded arrays or ad hoc samplers.A numerical yield score cannot establish scientific validity, so execution traces are essential for separating legitimate errors from fabricated or incomplete workflows.
5 Conclusion … A.2 Glossary
Collider-Bench evaluates whether coding agents can reproduce collider-physics analyses despite incomplete public information, using executable simulation-and-selection workflows and quantitative performance metrics. The surrounding primer and glossary define the recasting workflow, collider-analysis concepts, and simulation quantities needed to interpret these tasks.
- 5 Conclusion: Public information does not uniquely determine the correct reconstruction, requiring agents to make reasonable choices to fill missing analysis details.This underdetermination is a defining challenge of the benchmark.
- 5 Conclusion: Collider-Bench evaluates six off-the-shelf agents on ten tasks spanning four LHC analyses, finding that most completed tasks but none reliably matched a physicist-in-the-loop.Performance improved when tasks were restricted to well-scoped subsets, and the benchmark exposes recurring agent pitfalls.
- A Brief Primer on Collider Physics: Collider analyses reconstruct detector signals into physics objects and select events using kinematic criteria such as energetic photons, jets, and missing transverse momentum.An event is a reconstructed proton–proton collision record containing measured object properties.
- A Brief Primer on Collider Physics: Searches compare observed event counts with Standard Model background predictions while testing additional signal contributions from hypothetical new-physics models.Compatibility with background-only expectations can constrain signal models, whereas excesses may indicate a signal.
- A Brief Primer on Collider Physics: A collider prediction proceeds from a Lagrangian through hard-scattering, parton-shower, hadronization, and detector-response simulations to produce observables.This multi-stage Monte Carlo pipeline is decomposed by energy scale.
- A Brief Primer on Collider Physics: A recast reconstructs a published search by implementing object definitions and selections, generating signal events, simulating the detector, and estimating final yields.The full internal analysis code is usually unavailable, so the implementation relies on public information.
- A.1 Recasting toolbox: The recasting toolbox provides command-line wrappers for reading papers, querying HEPData, and accessing the public simulation stack needed to reproduce analyses.Agents use these fixed tools as they would in a real recasting workflow.
- A.2 Glossary: Key glossary quantities include signal regions defined by reconstructed-object and kinematic cuts, acceptance times efficiency as the surviving event fraction, and Monte Carlo tools for generating signal samples.The glossary also defines detector simulation, fast detector simulation such as Delphes, and recast validation against known yields or distributions.
B Benchmark Construction Details · B.1 Event-selection specification to code
Collider-Bench requires agents to translate published event-selection criteria into executable analysis logic. Figure 5 illustrates this specification-to-code step.
- B.1 Event-selection specification to code: Agents must convert event-selection criteria into executable analysis logic.This conversion is the required translation step in Collider-Bench.
- B.1 Event-selection specification to code: The event-selection criteria originate from the published analysis description.The benchmark uses the published description as the source for the selection requirements.
- B.1 Event-selection specification to code: The translation step connects published analysis descriptions to executable code.Figure 5 illustrates this connection in Collider-Bench.
- B.1 Event-selection specification to code: The required output of the translation is executable analysis logic.The agent’s task is not merely to restate the criteria, but to express them as executable logic.
- B.1 Event-selection specification to code: Figure 5 illustrates how agents perform the event-selection translation.The figure presents the benchmark’s specification-to-code requirement.
- B.1 Event-selection specification to code: Event-selection specification to code is a required component of Collider-Bench.The benchmark explicitly requires agents to implement published selection criteria as analysis logic.
B.2 Task Interface … B.5 Agent Workspace and Sandboxing
Collider-Bench gives agents a paper, task instructions, tools, and a null-filled YAML template, then requires them to build and run their own simulation-and-selection pipeline to predict signal yields. The workspace and sandbox preserve controlled information access while recording artifacts and execution traces, and the yield formulation separates distribution shape from absolute normalization.
- B.2 Task Interface: Each task supplies a target paper, task prompt, tool documentation, and null-filled YAML output template specifying the signal benchmark, observable, regions, and bins.Hidden reference yields and evaluator code are excluded from the agent workspace.
- B.2 Task Interface: The agent must generate benchmark events, infer object definitions and selection cuts from the paper, apply the analysis, and histogram surviving events without modifying template bin edges.Predictions must come from the agent’s own simulation and analysis pipeline rather than copied values from papers, figures, tables, or HEPData.
- AGENTS: The workspace provides papers, results, efficiency files, command-line tools, Python libraries, and simulation resources for implementing and executing the recast.Agents are instructed to use bin/run-analysis rather than running analysis.py directly, and to invoke tool documentation when exact flags, schemas, or caveats are needed.
- B.3 Representative Output Template: The representative output template requires replacing only null dependent-variable values while preserving metadata, file structure, and predefined bin edges, including overflow events in the rightmost bin.The example targets T5Wg_1750_1700 at 13,000 GeV and 35.9 fb−1 with four S_T^gamma bins.
- B.4 From Shape to Absolute Event Yields: Predicted yield in each bin combines a normalized selected-event distribution with an absolute event yield determined by luminosity, cross section, branching fractions, and acceptance times efficiency.Acceptance times efficiency is estimated from generated events that survive detector simulation, reconstruction, selection, and bin assignment, using weighted sums when appropriate.
- B.4 From Shape to Absolute Event Yields: Shape and normalization expose different failure modes: selection or binning errors distort the distribution, whereas luminosity, rate, branching, or weight errors alter the total scale.Published rates may additionally require higher-order cross sections, filter efficiencies, generator-level branching assumptions, or model-specific decay tables.
- B.5 Agent Workspace and Sandboxing: Each run starts in a fresh, writable workspace where the agent can inspect files, write and execute code, invoke tools, run simulations, revise its analysis, and submit a prediction.Hidden references, evaluator code, private metadata, other tasks, and prior artifacts are not mounted; submitted yields must originate from the agent’s recast.
- B.5 Agent Workspace and Sandboxing: The benchmark records task configuration, workspace contents, generated artifacts, final outputs, and a session.jsonl execution trace containing commands, tool calls, intermediate outputs, and state changes.The trace is not used for the primary numerical score but supports downstream qualitative evaluation and provenance analysis.
C Experimental Details … C.3 Physicist-in-the-loop Baseline
Collider-Bench standardizes agent execution through a shared sandbox and run protocol, while benchmarking a supervised physicist-in-the-loop baseline. The baseline shows that expert guidance is most valuable for scientific judgment about conventions, normalizations, and discrepancies.
- C.1 Agent Systems: Agents are evaluated as complete language models paired with agentic coding scaffolds, including Claude Code, Codex CLI, and ForgeCode.The benchmark evaluates these systems rather than isolated language models.
- C.1 Agent Systems: Every agent uses the same sandbox controller, task specification, and tool documentation without model-specific prompt tuning.The controller creates fresh workspaces, mounts inputs, launches the selected CLI, records traces, and collects final artifacts.
- C.2 Run Protocol: Each run starts from an empty workspace with the target paper, TASK.md, TOOLS.md, and a null-filled YAML output template.Runs end when the agent stops, the 2.5-hour wall-clock budget expires, or the sandbox reports a fatal error; the existing template is submitted.
- C.2 Run Protocol: A submission passes only if its YAML parses, has the expected bins, contains finite non-negative values, and passes the provenance audit.Only structurally passing submissions receive relative-L2 fidelity and Accτ scores.
- C.2 Run Protocol: Syntactically valid but null, zero, or unreadable submissions remain in per-model averages, while FABRICATED runs are excluded from per-task aggregates.This preserves actual pipeline failures in headline results while removing values that fail provenance validation.
- C.3 Physicist-in-the-loop Baseline: The supervised baseline pairs Claude Code with a domain expert while following the same external-recast simulation-and-selection pipeline as autonomous tasks.The agent performs most coding, including event loops, plotting, cutflows, cross-section lookups, and yield-comparison utilities.
- C.3 Physicist-in-the-loop Baseline: The human supervisor supplies the next analysis step, checks physical plausibility, and resolves domain-specific choices about definitions, cross sections, weights, and discrepancies.The human role is supervisory rather than implementational.
- C.3 Physicist-in-the-loop Baseline: Expert guidance lets a capable agent carry much of the coding burden, but scientific judgment about conventions, normalizations, and consequential discrepancies remains the bottleneck.This is the central difficulty illustrated by the physicist-in-the-loop baseline.
D Additional Results
This appendix presents additional results on run completion, yield and shape accuracy, model performance, cost–accuracy tradeoffs, and overlays against published results.
- Additional Results: The appendix includes run completion rates, relative-L2 errors for absolute binned yields and unit-normalised shapes, performance on shape tasks, cost–accuracy tradeoffs, and overlays against published yields and shapes.These results are presented in Figures 6–9 and Tables 2–3.
D.1 Tool Ablation Test
Disabling Delphes for GPT-5.5 showed that detector-tool necessity depends on the task. Agents sometimes constructed adequate approximations, but CMS-SUS-16-047 degraded more clearly without detector simulation.
- Ablation setup: Disabling Delphes left the papers, prompts, templates, and analysis environment unchanged while removing the standard detector-simulation step.The ablation reran GPT-5.5 simulation tasks with the remaining interface unchanged.
- Task dependence: For several tasks, scores with and without Delphes were comparable, indicating that strong agents can sometimes construct adequate detector approximations.The effect was explicitly task dependent.
- Detector-sensitive analyses: CMS-SUS-16-047 tasks degraded more clearly without Delphes, suggesting greater sensitivity to detector-level modeling, missing-momentum response, or bin migration.These harder tasks could not be replaced as easily by lightweight approximations.
- Interpretation: The ablation shows that domain-tool necessity is task dependent: approximate substitutes may suffice in simpler settings, whereas validated detector toolchains remain important in sensitive analyses.The benchmark therefore tests whether agents can decide when an approximation is appropriate.
E LLM Judge
A separate fixed LLM judge audits each run for provenance and qualitative trajectory, while recovered outputs can trigger automatic rescoring. Aggregate results exclude only fully invalidated FABRICATED runs, and the trajectory narrative does not affect numerical scores.
- Audit inputs: The judge examines submitted results, hidden references for leakage detection, workspace artifacts, and a 50k-character structured session-log extract.The full session trace is not provided to the judge.
- Audit outputs: The audit returns provenance findings and an evidence-based narrative covering planning, tool use, scientific judgment, and stuck points.The narrative is recorded for qualitative review but does not enter the numerical score.
- Rescoring: Recovered bin values that agents computed but omitted from the template automatically trigger rescoring against those values.This allows the benchmark to evaluate recoverable outputs rather than treating omitted template entries as final submissions.
- Aggregate handling: Aggregate tables exclude only runs the judge fully invalidated as FABRICATED, while rescored runs use the judge’s recovered values.Thus, provenance auditing affects inclusion and submitted values in the aggregate results.