MOF-VERIFY: A Failure-Aware Agentic Harness for MOF Hypothesis Verification

1 Introduction

• Structural aliases, condition-dependent synthesis, incomparable evidence across studies, and hypotheses requiring calculations make plausible LLM answers insufficient for reliable verification.These challenges span material grounding, experimental-context recovery, evidence acquisition, and computational verification.Source: §1 · pp. 1–2
• End-to-end accuracy can conflate distinct failures in structural retrieval, synthesis recovery, evidence sufficiency, and computational verification, although they require different interventions.The benchmark exposes failures at distinct pipeline stages to guide targeted interventions.Source: §1 · p. 2
• For T-MOF-1–3, a 3 × 3 design crosses task families with closed-book, retrieval-enabled, and oracle-evidence settings to localize where and why failures arise.Systems predict Yes, No, or Uncertain; T-MOF-4 uses a separate computational protocol.Source: §1 · p. 2
• The diagnostic benchmark separates four verification tasks: structural grounding, synthesis-information verification, evidence sufficiency, and MLIP-based computational verification.These correspond to T-MOF-1 through T-MOF-4.Source: §1 · p. 2
• MOF-VERIFY combines specialized structural, literature, evidence-sufficiency, and computational modules with Verdict Routing to integrate outputs and determine the final verdict.The harness targets the diagnosed bottlenecks before issuing a verdict.Source: §1 · p. 2

2 Related Work

• Prior MOF LLM studies extract synthesis conditions and properties [Zheng et al., 2023, Kang et al., 2025, Pruyn et al., 2025], automate simulation, and couple hypotheses to MLIP validation [Ahn et al., 2026].Other efforts connect literature, structural databases, and computational tools across MOF discovery.Source: §2 · p. 3
• Scientific and MOF-specific benchmarks cover reasoning, knowledge, extraction, database querying, prediction, experiment design, and tool use, but largely assess final task performance.They generally do not trace incorrect verdicts to specific Co-Scientist pipeline stages.Source: §2 · p. 3
• Existing MOF systems and benchmarks do not systematically localize failures across the full verification pipeline, including identity ambiguity, condition dependence, cross-study comparison, and computational limits.The paper addresses this gap with a controlled diagnostic benchmark and a failure-aware harness.Source: §2 · p. 3

3.1 Benchmark Design for MOF Hypothesis Verification

• T-MOF-1–4 isolate structural retrieval, synthesis-condition verification, literature evidence sufficiency, and MLIP-based computational verification, respectively.T-MOF-1–3 use Yes, No, and Uncertain verdicts under closed-book, retrieval-enabled, and oracle-evidence settings; T-MOF-4 requires workflow execution.Source: §3.1 · pp. 3–4
• The task controls require grounding material names to CSD refcodes, checking exact literature-reported synthesis routes, and evaluating contested propositions across studies.T-MOF-1 includes decoy refcodes; T-MOF-2 perturbs one route component; T-MOF-3 includes 117 one-sentence propositions.Source: §3.1 · p. 4
• Each benchmark task records the evidence needed for its reference verdict, while core items exclude questions answerable by a fixed no-tool LLM baseline.Evidence anchors are CSD refcodes, literature records with DOI, expert-specified decisive conditions, or fixed MLIP specifications.Source: §3.1 · pp. 4–5
• Table 2 reports Macro-F1 across T-MOF-1–3 evidence-access settings and T-MOF-4 Closed-book, with MOF-VERIFY comparison scores for the rule-based tasks.The caption reports identical MOF-VERIFY scores across models of 78.31 for T-MOF-1 and 56.97 for T-MOF-4.Source: Table 2 · p. 5

3.2 Diagnosing MOF Verification Bottlenecks with T-MOF-1–4

• The diagnosis combines controlled evidence-access comparisons with expert review of incorrect predictions to identify concrete task-specific failure modes.T-MOF-1 and T-MOF-2 use rule-based rationale screening, while T-MOF-3 uses an LLM judge; categories may overlap.Source: §3.2 · pp. 5–6
• Across T-MOF-1–3, RAG improves Macro-F1 over Closed-book by 23.64 points on average, and Oracle adds 12.59 points.T-MOF-1 gains most from RAG and Oracle; T-MOF-3 has negative average RAG access gain and remains difficult with Oracle evidence.Source: §3.2 · p. 6
• The task-specific failure-rate table reports percentages among incorrect predictions, averaging Closed, RAG, and Oracle rates without quarantined cases.The diagnosis defines structural, synthesis, and evidence-insufficiency rates over the corresponding incorrect predictions; over-commitment measures binary verdicts on gold-Uncertain cases.Source: §3.2 · pp. 6–7, Table 3 · p. 7
• MOF-VERIFY’s T-MOF-1–3 diagnostic rates average 94.87% for structural errors, 93.77% for synthesis errors, and 64.66% for evidence-insufficiency errors.Over-commitment affects 81.18% of gold-Uncertain cases; diagnostic rates remain high across Closed, RAG, and Oracle settings.Source: §3.2 · p. 7

4 MOF-VERIFY: A Failure-Aware Agentic Harness

• MOF-VERIFY uses a frozen base LLM without additional training or fine-tuning, orchestrating specialized modules that target diagnosed bottlenecks.Each module acquires, validates, or integrates MOF-specific evidence before a scientific verdict.Source: §4 · p. 7

4.1 Workflow

• The workflow processes each hypothesis through IR, SE, LE, ES, and CE, then integrates provenance-aware outputs in a deterministic router.IR resolves material identity; SE retrieves a trusted deposited structure; LE extracts synthesis conditions; ES checks evidence reliability and sufficiency; CE computes quantities not established by documents.Source: §4.1 · p. 8, Figure 1 · p. 8
• IR and SE use curated identity resolution and deterministic lookup to link a canonical material identity to a trusted, provenance-carrying structure.SE ranks experimental records first and requires the crystallographic file to match its vetted digest.Source: §4.1 · p. 8
• LE extracts synthesis records from verified local documents, validates them against a schema, and reports conflicts or unavailable sources rather than answering from memory.When the sample identity cannot be resolved, it retains queried conditions with the verbatim reporting sentence.Source: §4.1 · pp. 8–9
• ES flags evidence conflicts and gaps, while VR requires complete provenance, trusted sources, and valid computation for decisive verdicts.When evidence is insufficient or these checks fail, VR abstains or returns Uncertain instead of forcing a binary label.Source: §4.1 · p. 9
• CE converts requested quantities into executable specifications and returns computed properties with convergence records or typed refusals.For T-MOF-4, valid results yield Yes, non-convergence yields Re-run, and unsupported tasks yield Human-review.Source: §4.1 · p. 9

4.2 Performance Analysis

• 62.82 average Macro-F1 across T-MOF-1–3 improves over Closed-book and RAG by 31.80 and 8.16 points, respectively, approaching the Oracle upper bound of 67.26.Performance remains task-dependent: T-MOF-1 reaches 78.31 Macro-F1 and T-MOF-3 improves over RAG by 9.38 points on average, while T-MOF-2 remains a residual bottleneck. On T-MOF-4, CE raises average performance from 5.74 under direct LLM inference to 56.97.Source: §4.2 · p. 9
• Targeted modules substantially mitigate diagnosed bottlenecks while revealing where further improvement is needed.The performance analysis reports that synthesis-condition extraction remains a residual challenge.Source: §4.2 · pp. 9–10

• A diagnostic benchmark decomposes MOF hypothesis-verification failures into knowledge, retrieval, and computation bottlenecks.Source: §5 · p. 10
• The identified failures persist even with improved evidence access, motivating targeted intervention beyond retrieval alone.Source: §5 · p. 10
• A failure-aware agentic harness improves verification by targeting the diagnosed bottlenecks.Source: §5 · p. 10

A.1 Sampled example hypothesis of T-MOF-1-4

• The harness rejects OCUNAC as the trusted UiO-66 structure because it is MIL-101, a Cr-based framework with a different topology.Source: §A.1 · p. 13
• IRMOF-13 and IRMOF-14 are not identical structures despite their shared Zn4O/PDC composition and pcu topology.IRMOF-13 is a simulated, doubly interpenetrated trigonal R¯3m model; IRMOF-14 is an experimentally refined, non-interpenetrated cubic Fm¯3m structure.Source: §A.1 · p. 13
• The Co2(m-DOBDC) methyl-iodide treatment was reported at 100 °C for 48 h, while 60 °C applies to formaldehyde treatment.Source: §A.1 · p. 13
• The reported Zn-framework MOF-1 route matches the hypothesis: H6L and Zn(NO3)2·6H2O in DMF were heated at 78 °C for 3 days.The Experimental Section specifies 8 mL DMF and a 15 mL Teflon reactor.Source: §A.1 · p. 13
• Same-composition MOF glasses can have different glass-transition temperatures when formed under different pressures.The ZIF-62 glass quenched at 60 MPa had a significantly lower glass-transition temperature than the same-composition glass formed at ambient pressure.Source: §A.1 · p. 13
• Disappearing sharp PXRD Bragg peaks leave atomic-scale order uncertain because decomposition and vitrification differ in short-range order.Source: §A.1 · p. 14
• MOF-5 converges in 70 steps at fmax = 10−6 eV/Å, whereas a one-step UiO-66 relaxation is non-converged and should be rerun.The same tight threshold does not converge for UiO-66 within 80 steps.Source: §A.1 · p. 14

A.2 Complete Results for T-MOF-1–3

• Table 4 reports class-wise F1 for T-MOF-1–3 across three evidence-access settings, while Table 5 reports MOF-VERIFY results for T-MOF-2–3.Source: §A.2 · p. 14, Table 4 · p. 15, Table 5 · p. 15

A.3 The Example Hypothesis Traced through Our MOF-VERIFY Workflow

• A stage that cannot ground its step emits a typed failure instead of passing unsupported information forward.Examples include an unrecognized material, unverifiable file, missing literature record, or insufficient evidence.Source: Figure 2 · p. 16
• Verdict Routing returns the verdict corresponding to the typed failure.Source: Figure 2 · p. 16
• The example traces one hypothesis through the harness, showing how grounding failures avoid an unsupported conclusion.Source: Figure 2 · p. 16

A.4 Failure Classes and Corresponding MOF-VERIFY Components

• Table 6 maps each identified failure class to the MOF-VERIFY component that addresses it.Source: Table 6 · p. 16, §A.4 · p. 16

A.5 Evaluation Prompts and Task-Specific Decision Procedures

• T-MOF-1–3 use closed-book, retrieval-enabled, and oracle-evidence settings, while T-MOF-4 follows the computational-verification path; benchmark gold labels are excluded from model inputs.Source: §A.5 · p. 16

A.5.1 Closed-book setting (T-MOF-1–3)

• In closed-book T-MOF-1–3, models use only internal knowledge, return a fixed JSON verdict format, and should abstain when uncertain.Source: §A.5.1 · p. 16

A.5.2 Retrieval-enabled setting (T-MOF-1–3)

• In retrieval-enabled T-MOF-1–3, models use the question and retrieved excerpts only, and should answer Uncertain or Human-review when excerpts do not support a decision.The prompt allows up to eight retrieved chunks and forbids claiming to have read a complete paper.Source: §A.5.2 · p. 17

A.5.3 Oracle-evidence setting for structural grounding (T-MOF-1)

• For structural-grounding T-MOF-1, oracle packets may combine source-paper or SI excerpts, licensed CSD records, and CIF text, and inclusion alone does not establish a claim's correctness.Source: §A.5.3 · p. 17

A.5.4 Oracle-evidence setting for synthesis-condition and evidence-sufficiency verification (T-MOF-2–3)

• For synthesis-condition and evidence-sufficiency T-MOF-2–3, oracle evaluation uses the item's source documents and directs models to answer Uncertain when evidence is not decisive.Source: §A.5.4 · p. 18

A.5.5 MOF-VERIFY procedures (T-MOF-1–4)

• In MOF-VERIFY structural grounding, Identity Resolution and Structural Evidence resolve identity and validate the trusted structure before Verdict Routing deterministically compares the claim with the record.Source: §A.5.5 · p. 19
• For T-MOF-2, Literature Evidence first checks the structured synthesis record and sends undecided items to paper reading; T-MOF-3 uses the paper-reading procedure with full available paper text.Source: §A.5.5 · p. 19
• For computational T-MOF-4, Computational Evidence parses the calculation and convergence controls, runs SevenNet-based relaxation, and Verdict Routing derives the verdict from the typed result.T-MOF-4 is not evaluated under the three evidence-access settings.Source: §A.5.5 · p. 19