Source-linked AI summary
AutoXRD: Autonomous LLM Agents and Comprehensive Evaluation for Powder Diffraction Analysis
Yuetong Wu, Maojun Sun
TL;DR
Reliable powder-XRD automation must coordinate scientific interpretation, software execution, parameter coupling, and physical validity. AutoXRD addresses this with evidence-gated stepwise refinement and XRDBench’s complementary diagnostic and end-to-end tracks; across models, diagnostic performance exceeds autonomous execution, while recurring failures remain in scientifically constrained workflow control.
Problem
Existing tool-using agents and XRD studies lack general scientific guarantees and systematic evaluation of complete, physically valid workflows.
Method
AutoXRD organizes analysis as evidence-gated stepwise refinement, while XRDBench evaluates bounded diagnostic tasks and executable end-to-end workflows.
Results
61.9 on XRDBench-QA versus 53.7 on XRDBench-E2E shows that diagnostic competence does not reliably translate into autonomous execution.
Takeaways & Limitations
Reliable diffraction agents must jointly expose numerical fit, physical validity, and reproducible execution evidence rather than produce merely plausible reports.
Takeaways & Limitations
A scalar fit index cannot establish structural correctness because its value depends on data quality and uncertainty conventions.
Abstract
from arXiv · showhide
Powder X-ray diffraction (XRD) is central to materials characterization, yet reliable end-to-end automation remains challenging. An XRD agent must interpret diffraction evidence, operate refinement software, manage coupled parameters in a defensible order, and distinguish numerical improvement from physical validity. In this paper, we propose AutoXRD, an autonomous large language model (LLM) agent framework that organizes powder-XRD analysis as stepwise refinement, grounds actions in observed evidence, and applies deterministic crystallographic and physical checks before accepting results. We further introduce XRDBench with two complementary tracks. XRDBench-QA contains 100 bounded diagnostic tasks that isolate scientific reasoning and decision-making, whereas XRDBench-E2E contains 34 executable workflows that test whether agents can compose these capabilities into complete analyses requiring file inspection, crystallographic-software execution, iterative refinement, evidence preservation, and reporting. We evaluate ten recent LLMs across 1,340 model--task runs. Models average only 57.8 out of 100, falling from 61.9 on XRDBench-QA to 53.7 on XRDBench-E2E. They perform best on refinement-history assessment and result acceptance, but remain substantially weaker on refinement-action selection, phase quantification, indexing, and Rietveld refinement. GPT-5.6 Sol achieves the highest overall score of 81.1, GPT-5.6 Terra the highest XRDBench-E2E point estimate of 81.0, and GPT-5.6 Luna the best score--cost trade-off. Ablations show that all six AutoXRD components consistently improve performance, supporting the framework design. Finally, execution-trace analysis reveals recurring failures in coupled-parameter control, quantitative reasoning, evidence preservation, and workflow termination, motivating stronger scientific constraints, uncertainty-aware decisions, and more efficient planning.
1 Introduction
AutoXRD frames powder-XRD analysis as evidence-gated, stepwise scientific refinement rather than unconstrained tool use. XRDBench separates diagnostic reasoning from executable end-to-end analysis, exposing a substantial gap between the two.
- XRDBench: 100 bounded XRDBench-QA tasks and 34 XRDBench-E2E workflows test complementary capabilities from scientific decisions to executable, evidence-preserving analyses.QA isolates reasoning, whereas E2E requires file inspection, numerical execution, validation, and auditable delivery.
- Evaluation findings: 61.9 on XRDBench-QA versus 53.7 on XRDBench-E2E shows that diagnostic competence does not reliably transfer to autonomous execution.The two track scores are only moderately correlated, with r = 0.56 and ρ = 0.65.
- Evaluation findings: Models perform best on refinement-history assessment and result acceptance but remain weaker on refinement-action selection, phase quantification, indexing, and Rietveld refinement.These weaknesses identify decision-making and complete refinement as major remaining bottlenecks.
- AutoXRD framework: AutoXRD combines skill-guided workflows, executable crystallographic backends, and deterministic validation within an auditable refinement process.Candidate actions are supported by evidence and checked for crystallographic and physical validity before acceptance.
2 Background and Related Work
Powder-XRD refinement is difficult because one-dimensional profiles and coupled parameters permit multiple plausible explanations. Prior automation provides numerical engines and specialized workflows, while AutoXRD focuses on constraining and evaluating the sequence of scientific decisions.
- XRD refinement: Rietveld refinement fits complete diffraction profiles, but coupled structural, specimen, and instrument parameters require staged release rather than unconstrained curve fitting.The profile compresses reciprocal-space information and can contain overlapping effects from background, orientation, size, strain, and mixtures.
- Failure modes: A lower weighted-profile factor does not establish structural correctness because fit values depend on data quality and uncertainty conventions.Explicit checks of residual structure, parameter correlations, chemistry, bounds, and unresolved peaks are therefore needed.
- Related work: Existing automation spans crystallographic engines, high-throughput wrappers, expert systems, reinforcement learning, black-box optimization, and learned parameter prediction.These approaches automate numerical capabilities or parameter selection across different parts of the workflow.
- Related work: Phase-identification and end-to-end pipelines automate database matching, structure generation, search, refinement, or quantitative phase fitting, but remain distinct from constraining an agent’s decision sequence.AutoXRD positions its contribution as complementary to these systems.
- AutoXRD’s distinction: AutoXRD makes action schemas, falsifiable predictions, stage order, coupled-parameter exclusions, physical checks, and rejected steps part of the executable protocol.This distinguishes it from generic language-agent loops and supports evaluation under sharp validity constraints.
3 Problem Formulation
AutoXRD formalizes refinement as evidence-bearing transitions between complete fit states, with structured actions and hard acceptance criteria separating numerical improvement from scientific admissibility.
- 3.1 Refinement States and Structured Actions: A refinement state records fit quantities, residual features, physical violations, complexity or correlations, and backend output files.
- 3.1 Refinement States and Structured Actions: AutoXRD supports validated actions defined by an action kind, refinement stage, parameter subset, optional bounds, cited evidence, and falsifiable predictions.
- 3.1 Refinement States and Structured Actions: Native code-agent actions can read, create, edit, or execute task files, with broader actions retained and evaluated through execution, artifact, and physical gates.
- 3.1 Refinement States and Structured Actions: A deterministic backend produces candidate states, but acceptance separately asks whether optimization improved numerically, the predicted feature changed, and the child state is admissible.
- 3.2 Refinement Decision Graph: The decision graph represents fit states as vertices, accepted actions as E+, and rejected actions as E−.
- 3.2 Refinement Decision Graph: Hard admissibility H and prediction support M gate transitions before utility is considered.
- 3.2 Refinement Decision Graph: The acceptance rule requires H, M, and positive utility simultaneously; utility cannot compensate for failed admissibility or prediction support.
4 AutoXRD
AutoXRD combines specialized XRD procedures, executable crystallographic backends, and deterministic checks so that language models plan and explain analyses without serving as the optimizer or final scientific authority.
- The procedure library provides ten XRD skills spanning quality control, structure validation, refinement, residual diagnosis, result acceptance, and scientific review.
- Language models select context-dependent hypotheses, while crystallographic engines perform nonlinear optimization and deterministic code enforces exact invariants.
- Residual features include robust standardized residuals, autocorrelation, angle-dependent bias, contiguous extreme-residual regions, and unexplained peaks.
- A proposed action must predict the relevant feature change; a lower Rwp is insufficient when the predicted feature does not improve.
- The PCR validator freezes catalogued codeword groups, releases only permitted selectors, validates stage and constraints, and never overwrites the source file.
- Pre-execution checks reject out-of-stage or unbounded high-risk actions and constrain strongly correlated parameter combinations.
- Post-execution checks reject new physical violations, Rwp regressions above 2%, unexplained-peak increases above 0.02, correlations above 0.98, or failed predictions.
- Among candidates passing mandatory checks, transparent utility weights rank residual score, unexplained peaks, parameter count, and maximum correlation.
5 XRDBench
XRDBench evaluates XRD agents at two integration levels: bounded scientific decisions and executable workflows requiring implementation, validation, evidence, and delivery.
- XRDBench-QA isolates bounded reasoning through 100 tasks, while XRDBench-E2E tests 34 executable workflows requiring complete analyses.
- The two tracks evaluate the same process at different integration levels because endpoint-only evaluation hides causes and bounded questions cannot establish autonomy.
- 5.2 XRDBench-QA: XRDBench-QA includes 30 exact-option cases, 40 structured evidence-grounded reports, and 30 quantitative outcomes with reports.
- 5.2 XRDBench-QA: QA covers nine capabilities, with 70 deterministically generated tasks and 30 tasks based on experimental quantitative-phase-analysis and adapted phase-identification data.
- 5.2 XRDBench-QA: Objective metrics use baseline and oracle utilities, while quantitative recovery combines mechanism detection, normalized errors, phase-set F1, fraction error or closure, amorphous accuracy, and report quality.
- 5.3 XRDBench-E2E: The E2E track spans eleven task families, including phase analysis, prediction, indexing, peak and microstructure analysis, structure determination, retrieval, and Rietveld refinement.
- 5.3 XRDBench-E2E: Each E2E workspace supplies tools and crystallographic software, and agents must produce final_report.md, executed code, and evidence subject to execution and physical gates.
- 5.3 XRDBench-E2E: Runs record trajectories, commands, artifacts, timing, model turns, tool calls, and tokens for audit.
6 Experiments
The experiments evaluate ten language models under bounded QA and end-to-end execution protocols, with research questions covering capability, composition, integration-dependent failure, and efficiency.
- Ten models are evaluated under separate QA and E2E limits, with GPT-5.6 Sol providing post-execution scientific judgment.
- The study asks which XRD capabilities models master, whether they compose into reliable workflows, how failures change with integration, and how efficiency and components affect performance.
- Figure 3 orders models by overall score and displays Overall, XRDBench-QA, and XRDBench-E2E as separate bars.
- The evaluation comprises 1,340 runs across ten models and 22,135 model turns, with bootstrap intervals describing task variation rather than decoding stochasticity.
7 Results
Across XRDBench, models perform better on bounded diagnosis than executable workflows, with weaknesses concentrated in scientific action selection, indexing, refinement, and evidence-preserving completion. Component ablations and efficiency results show that verification improves performance, while score, cost, and latency trade off.
- Overall performance: The ten-model mean falls by 8.22 points, from 61.89 on XRDBench-QA to 53.67 on XRDBench-E2E, and seven models score lower end to end.This indicates that composing scientific decisions into executable analyses is harder than answering bounded questions.
- Scientific capability: Models score highest on refinement-history assessment (88.0) and result acceptance (81.0), but lowest on next-action selection (44.0) and quantitative phase analysis (33.0).They judge completed evidence more reliably than they control coupled parameters or recover phase fractions.
- End-to-end bottlenecks: Indexing scores 37.72 despite 90% scoring-gate passage, 80% complete deliverables, and 86.2 artifact validity, while its objective-metric and scientific-judge scores are 26.9 and 33.2.Agents often execute and document indexing workflows but recover inaccurate cells or HKL assignments.
- End-to-end bottlenecks: Rietveld refinement combines a low final score (48.58) with the lowest scoring-gate rate (71.25%) and only 50% complete deliverables.Workflow validity and evidence completion are bottlenecks alongside result quality.
- Rank transfer: QA and E2E scores correlate moderately (Pearson r=0.557; Spearman ρ=0.648), but model rankings change materially across tracks.DeepSeek-V4-Pro and Gemini 3.1 Pro Low gain 16.46 and 20.58 points, whereas three other models lose 21.8–27.6 points.
- Ablation and efficiency: Luna provides the strongest score–cost trade-off at 68.87 and $0.07 per task, but its 435-second duration exceeds Terra’s 239 seconds.Monetary and latency efficiency do not coincide.
- Failure analysis: Execution or physical gates fail in 46 of 340 runs (13.5%), while 117 runs (34.4%) fail at least one full-completion requirement.Missing or invalid code, backend outputs, and required evidence commonly prevent scoring or completion.
- Ablation and efficiency: Each of six tested components improves GPT-5.6 Luna’s XRDBench-QA performance, with result-acceptance checks and final scientific review gaining 17.5 and 17.1 points.Domain procedures add 2.5 points, while pattern quality control, residual reasoning, and staged refinement add 1.1–1.6 points.
8 Discussion
XRDBench reveals complementary weaknesses: end-to-end workflows add implementation, validation, evidence-preservation, and delivery demands, while models can accept invalid refinements or omit verification artifacts. Reliable agents therefore need joint evaluation of numerical fit, physical validity, and reproducible execution evidence.
- End-to-end workflows expose failures that bounded questions do not, including implementation, backend operation, evidence preservation, physical validation, and delivery.
- Models may accept an invalid refinement when one fit statistic improves despite residual-pattern deterioration, or produce convincing reports without verifiable output files and intermediate results.
- Reliable XRD agents must evaluate numerical fit, physical validity, and reproducible execution evidence together.
9 Conclusion
AutoXRD combines planning, crystallographic tools, and deterministic validation, while XRDBench tests both diagnostic reasoning and end-to-end workflows. Across 1,340 runs, diagnostic competence only moderately predicts autonomous performance, underscoring the need for evidence and invariants behind conclusions.
- AutoXRD combines LLM planning, crystallographic tools, and deterministic validation, while XRDBench evaluates diagnostic reasoning and end-to-end workflows.
- Across ten models and 1,340 runs, indexing was executable but inaccurate, while Rietveld refinement was limited by completion and validity.
- Reliable diffraction agents must expose the evidence and invariants supporting each conclusion, not merely produce plausible reports.
Ethical Considerations
AutoXRD is intended as decision support rather than expert certification because incorrect autonomous diffraction conclusions can affect materials decisions and downstream experiments. Users remain responsible for provenance, safety, calibration, licensing, and independent scientific review.
- Incorrect autonomous diffraction conclusions can affect materials decisions and downstream experiments.
- AutoXRD provides decision support rather than expert certification and exposes uncertainty and validation failures.
- Users remain responsible for provenance, safety, calibration, licensing, and independent scientific review.