Source-linked AI summary
MLIP Detective: Active Failure Mode Discovery Beyond Benchmark Scores for Machine-Learning Interatomic Potentials
Ryuhei Okuno, Nontawat Charoenphakdee, Kaoru Hisama, Yuta Tsuboi
TL;DR
Benchmarks reproducibly evaluate u-MLIPs but cover only a limited region of configurational space, while extending coverage requires costly reference calculations. MLIP Detective uses physics-informed active search from benchmark evidence and surfaced an adsorption-energy anomaly associated with mixed PBE/PBE+U training data.
Problem
Benchmarks evaluate predefined tasks covering only a limited region of configurational space, and expanding coverage typically requires costly DFT reference calculations.
Source: §1 · p. 1Method
Results
MLIP Detective surfaced an adsorption-energy anomaly associated with selectively applied PBE+U, with mixed PBE/PBE+U reference energies identified as its likely origin, consistent with prior reports [18] [47].
Source: §3.1 · pp. 4–5Takeaways & Limitations
The framework can complement benchmark evaluation by preparing candidate failures and verification materials for independent human assessment.
Source: §1 · p. 2Search remains bound by the inspection specification, and escalated findings require costly human verification.
Source: §4 · p. 6
Abstract
from arXiv · showhide
Universal machine-learning interatomic potentials (u-MLIPs) aim to generalize across diverse configurations. Benchmarks enable reproducible evaluation but may not expose failures outside their predefined scope. Here, we show that physics-informed search can complement benchmark-based evaluation by uncovering hidden failure modes. We introduce MLIP Detective, an agentic framework for active failure mode discovery. Starting from benchmark evidence, MLIP Detective generates falsifiable, physics-informed failure hypotheses, screens them with inexpensive simulations, and escalates only the most suspicious cases to human experts together with proposed verification protocols. Without issue-specific prompting, MLIP Detective identified and characterized a systematic anomaly in MACE-MPA-0: the model predicted some relaxed adsorbate-surface systems involving O- or F-containing adsorbates to be higher in energy than their corresponding separated fragments. Using cross-model comparisons, MLIP Detective further inferred a likely training-data origin for the anomaly, consistent with recent reports.
1 Introduction
Benchmarks provide reproducible evaluation of u-MLIPs but cover only a limited region of configurational space, and expanding them requires costly reference calculations. MLIP Detective addresses this gap with physics-informed active failure mode discovery that starts from benchmark evidence and prepares suspicious findings for human verification.
1 Introduction
- Benchmarks cover only a limited region of configurational space, while testing additional configurations typically requires costly DFT reference calculations.A model may score well yet behave unphysically elsewhere.Source: §1 · p. 1
- The challenge is to decide where to test by generating falsifiable failure hypotheses, screening them inexpensively, and reserving costly verification for promising candidates.The paper frames this open-ended, physics- and protocol-dependent task as one for LLM agents.Source: §1 · p. 1
- MLIP Detective starts from existing benchmarks and packages candidate failures with proposed causes, downstream impact, reproduction code, and DFT-ready structures for independent human verification.The paper presents physics-informed active failure mode discovery for MLIPs as largely unexplored.Source: §1 · p. 2
2 Methodology
MLIP Detective combines surrogate failure signals, LLM-generated test protocols, and candidate prioritization, with human verification reserved for suspicious cases. Its agents investigate hypotheses through independent probes and use disagreement and physical-constraint violations to decide whether to escalate.
2 Methodology
- Failure mode discovery seeks test protocols that expose candidate failures, which verifiers assess before confirmed failures are grouped into recurring, physically coherent patterns.The setup defines a target u-MLIP as a model mapping atomic configurations to quantities such as energy, forces, and stress.Source: §2 · p. 2
- The framework combines cross-model disagreement and physical constraints as surrogate signals, LLM-generated physics-informed protocols, and an acquisition strategy that prioritizes suspicious candidates.These components estimate failure likelihood without immediate costly verification.Source: §2 · p. 2
- MLIP Detective receives a target model, benchmark results, auxiliary u-MLIPs, and a human-authored inspection specification, then outputs a candidate failure mode with evidence and verification materials.The materials include a test protocol, reproduction code, and structures prepared for independent verification.Source: §2 · pp. 2–3
- Figure 2 uses Erel > 0 as a diagnostic flag for candidate failures; all 27 anomalous cases shown fall within the U-block.The U-block comprises designated transition metals paired with O- or F-bearing adsorbates, and the figure also compares predictions across model/channel configurations.Source: Figure 2 · p. 3
- The Detective Agent ranks hypotheses and launches Probe Agents, which independently test them on target and committee models; disagreement or specification violations can trigger human verification.Reports, including rejected hypotheses, enter evidence memory to inform later search iterations.Source: §2 · p. 3
3 Experimental Results
Two case studies use benchmark evidence to surface distinct MACE-MPA-0 anomalies: selective-PBE+U adsorption energetics and a desorption-path barrier that DFT does not reproduce. Cross-model evidence supports mixed-reference training as a likely origin of the adsorption anomaly.
3.1 Rediscovery of Adsorption-Energy Anomalies Associated with Selective PBE+U
- Across 14 metals × 8 adsorbates, Erel > 0 for 27/40 U-block combinations and 0/72 combinations outside it.Erel is the combined adsorbate–slab energy relative to the separated fragments.Source: §3.1 · p. 4
- Cross-model comparisons and SevenNet-Omni’s channel switch support mixed PBE/PBE+U reference energies as the likely anomaly origin, consistent with prior reports [18] [47].The human verifier judged additional DFT calculations unnecessary because closely related prior work had documented the issue.Source: §3.1 · p. 5
3.2 DFT-Verified Anomaly beyond Adsorption-Energy Benchmark Coverage
- CO/Cu(111), MACE-MPA-0’s most accurate benchmark system within 0.04 eV of experiment, nevertheless has a defective desorption path.The benchmark scores adsorption energy at the relaxed minimum, not the potential-energy surface along desorption.Source: §3.2 · p. 5
- A CI-NEB check from each model’s own relaxed endpoints retains MACE-MPA-0’s +0.209 eV barrier, while committee paths are monotonic.The check was part of human verification and tested whether the finding arose from a constrained scan.Source: §3.2 · p. 5
- Across six systems: +0.188 eV CO/Cu(111), +0.117 eV Pd(100)/NO, +0.095 eV Pd(111)/NO, and +0.031 eV Pt(111)/NO; Ru/Ir CO are clean.The CO/Cu(111) value is consistent with the +0.209 eV result under a different protocol.Source: §3.2 · p. 5
- On MACE-MPA-0’s CO/Cu(111) path, the barrier is +0.208 eV versus +0.022 eV in DFT; neither curve exceeds the desorbed reference on Matlantis-PFP.DFT single points were evaluated on the models’ converged paths.Source: §3.2 · p. 5, Figure 3 · p. 6
4 Limitations and future work
The framework’s search is bounded by its inspection specification, and escalated findings still require costly human verification.
4 Limitations and future work
- The search remains bounded by the inspection specification, which constrains the regions and failure modes agents can examine.Future work aims to extend the search to broader classes of simulation protocols and target models.Source: §4 · p. 6
- Escalated findings require costly human verification, which the authors identify as a remaining limitation.The authors propose automating quality control so experts can focus on the most impactful candidates.Source: §4 · p. 6
- The authors plan to extend the search across broader simulation protocols and target models.They also identify automated quality control of escalated findings as future work.Source: §4 · p. 6
A Related work
MLIP Detective builds on benchmark evaluation and ideas from active testing and model auditing, but targets physics-informed failure discovery rather than aggregate performance estimation or model improvement. Its protocol-level search and expert verification distinguish it from several related approaches.
Benchmarks for u-MLIPs
- MLIP benchmarks are standard tools for u-MLIP comparison, and MLIP Detective uses their results to search untested configurations rather than replace benchmark development.The paper describes the framework as complementary to benchmark development.Source: A Related work · p. 10
Active testing and efficient evaluation
- Classical active testing selects inputs from a static pool to estimate aggregate metrics, whereas MLIP Detective generates candidates beyond a fixed pool to discover failure modes.The cited approaches use surrogate loss estimates for acquisition, with LURE correcting selection bias [20] [30].Source: A Related work · p. 10
Proactive evaluation of generative models
- ProEval combines Bayesian performance estimation and failure discovery, while MLIP Detective uses disagreement, physical constraints, agent-generated protocols, and expert verdicts supported by costly DFT [16].ProEval uses transfer-informed Gaussian-process priors, Bayesian quadrature, and superlevel-set sampling.Source: A Related work · pp. 10–11
Adaptive probing of LLMs
- Fact-Audit and ProbeLLM adaptively probe language models; MLIP Detective applies a related probing paradigm to MLIPs with physics-informed protocols and DFT-supported expert verification [27] [17].Source: A Related work · p. 11
Red teaming; Uncertainty quantification and active learning for MLIPs
- Red teaming seeks failure-triggering inputs without a per-query DFT-like verification oracle, while MLIP active-learning methods use surrogate signals to acquire data for model updates [34] [29] [39] [35] [42] [9] [40].MLIP Detective instead evaluates a fixed model, and candidate findings may require costly DFT confirmation.Source: A Related work · p. 11
Adversarial auditing of MLIPs
- Proof-Carrying Materials searches compositional blind spots in a predefined feature space, while MLIP Detective searches physical test protocols that can expose relational failures [4].Its protocol-level search can localize anomaly conditions and produce DFT-ready verification protocols.Source: A Related work · p. 11
Agentic frameworks for atomistic simulation
- Other atomistic-simulation agents automate workflows or model construction, whereas MLIP Detective uses agents to form and test hypotheses about where a model fails [31] [15] [26].Source: A Related work · p. 11
B Framework and implementation details
The appendix details an agent workflow in which the Detective proposes and screens hypotheses, Probe Agents run independent protocols, and human experts verify escalated candidates. Pre-registration, physical checks, and a fixed escalation report structure support that process.
Roles and authority; Implementation
- The Detective Agent can reject or escalate candidates but cannot confirm failures; DFT is reserved for post-loop human verification.The agents operate as separate sessions, with Probe Agents spawned in parallel within a hypothesis.Source: B Framework and implementation details · pp. 12, 14
Hypothesis discipline and pre-registration
- The Detective pursues one hypothesis at a time, ranking candidates by practical importance and potential severity, and pre-registers expected failures before probes begin.Pre-registration commits expected checks on unevaluated configurations, constrains later review, and supplies the decision table for human verdicts.Source: B Framework and implementation details · p. 12
- An impact probe must test application-like conditions and can show that an exposed defect is screened away in practice, grounds to reject or deprioritize escalation.Source: B Framework and implementation details · p. 12
Evidence memory and information hygiene
- An append-only evidence memory records declarations, probe reports, rejections, and human verdicts; agents cannot read archived full reports from settled runs.The distilled memory is the only carry-over from past runs, limiting bias from old reports.Source: B Framework and implementation details · p. 12
Probe Agent briefs
- Probe briefs require identical protocols on the target and committee models, agent-designed diagnostics, and rerunnable reports with observables, disagreement, checks, and severity.If all models show the same anomaly, the protocol itself is suspect.Source: B Framework and implementation details · pp. 12–13
Evaluation and escalation
- Exact physical-constraint violations are decisive, whereas cross-model disagreement only raises suspicion and does not identify which model is wrong.The inspection specification directs agents to use physical constraints and intuition, not outlier status alone, to identify a suspect.Source: B Framework and implementation details · p. 13
- Escalation reports include the finding, application boundary, anomaly figure, reproduction script, DFT verification card with a decision table, and tested confounds.Source: B Framework and implementation details · p. 13
Inspection specification; Implementation
- The inspection specification defines eight physical constraints with rationales, label-free checks, and thresholds, while directing agents to skip properties guaranteed by the target architecture.The appendix says the run’s instructions and inspection specification did not identify a specific material system, dataset, or known target-model failure; agent roles used Claude Code and Claude Opus versions.Source: B Framework and implementation details · pp. 11, 13, 14
C Additional Experimental Result: Mid-range dimer attraction in SevenNet-Omni
SevenNet-Omni showed anomalous mid-range attraction for several dimers relative to its committee. The reported concern is bounded to dilute gas-phase thermodynamics, and definitive verification awaits more demanding calculations.
C Additional Experimental Result: Mid-range dimer attraction in SevenNet-Omni
- At R = 4.5 Å, SevenNet-Omni retains 17 % of its Sb2 well depth, 11 % of Mo2, and 9 % of W2, versus at most 4 % for committee models.The Cr2 dimer was excluded because the committee disagreed near equilibrium, leaving no label-free reference band; no committee model showed a comparable attractive tail.Source: C Additional Experimental Result:… · p. 14
- MLIP Arena’s aggregate diatomics metrics assess each model’s curve without comparing binding at a given distance to an external reference, so the tail can be overlooked.The benchmark candidate was seeded by those aggregate metrics, despite the anomaly being visible on careful inspection of individual curves.Source: C Additional Experimental Result:… · p. 14
- The concern is bounded to dilute gas-phase thermodynamics: bulk screening brings excess pair interaction within committee spread, while Sb2 retains 0.73 eV at 4.5 Å, or 7 kT at 1200 K.The residual binding distorts pair-association weights by orders of magnitude; escalation cited 32 two-atom single-point calculations as a low verification cost.Source: C Additional Experimental Result:… · p. 14
- The escalated report awaits expert verification; although 32 PBE single points are proposed, a definitive potential-energy surface verdict would require costlier multireference calculations.The authors deprioritized those calculations in favor of the more application-critical finding in Section 3.Source: C Additional Experimental Result:… · p. 14
D Limitations
The search is bounded by its inspection specification, and human verification limits findings per unit of expert time. The evaluation is small in scale, and publication-quality reporting still requires manual polishing.
D Limitations
- The search rarely proposes failure classes outside the inspection specification, so new classes currently require a human to revise it.Search quality is capped by the quality of that specification.Source: D Limitations · p. 14
- Each escalated candidate consumes expert attention and may require new reference calculations, limiting verified findings per unit of expert time.Source: D Limitations · pp. 14–15
- The framework's escalation reports target verification decisions, so figures and prose required manual reworking for publication.Automating publication-quality reporting is left to future work.Source: D Limitations · p. 15
- The experiments cover only a small number of search runs, target models, and failure modes, limiting the scale of the evaluation.The authors call for broader tasks, independent runs, quantitative metrics, comparisons with alternative search strategies, and component ablations.Source: D Limitations · p. 15