Source-linked AI summary
Evaluating a 4B open-weights local LLM for agentic DFT workflows: a literature reproducibility audit
Shambhu Bhandari Sharma
TL;DR
Hosted commercial models limit reproducibility, cost-effective iteration, and data privacy in agentic materials workflows. This work evaluates a fully local Qwen3:4B pipeline for literature auditing and DFT recomputation, finding that deterministic constraints support reliable assertions despite extraction and structural limitations.
Problem
Hosted, closed-weight models create reproducibility, economic, and data-privacy vulnerabilities, while corpus-wide methodological transparency for autonomous DFT recomputation remains untested.
Method
A fully local neurosymbolic workflow uses Qwen3:4B to extract code-dependent method parameters, translate them into SIESTA inputs, and execute gated DFT calculations across 57 studies.
Results
The extractor achieves 95.7% precision and 67.3% recall across 201 expert judgements, while complete GPU residency governs extraction quality and the audit identifies 19 of 57 studies as reproducible in principle.
Takeaways & Limitations
Open-weights models can drive autonomous first-principles workflows when every assertion is grounded verbatim and every proposal passes deterministic code gates.
Takeaways & Limitations
The pipeline cannot automatically detect composition mismatches, and structural reproducibility remains constrained when relaxed coordinates are unavailable and prototype rebuilding leads to different local minima.
Abstract
from arXiv · showhide
Agentic workflows in materials science relying on hosted commercial models face severe reproducibility, economic, and data-privacy constraints. To explore fully local agentic science, this work evaluates an open-weights Qwen3:4B model executing an autonomous scientific pipeline across varying hardware constraints. Applied to pentagonal two-dimensional materials, the system extracts parameters from unstructured text, translates them into density functional theory (DFT) inputs, and drives simulations to convergence under a strict neurosymbolic architecture where agents propose and deterministic code disposes. The workflow is guarded by verbatim text grounding and multi-pass inference unions to counteract hardware-induced structural collapse. Evaluated against 201 expert judgements, the extractor achieves 95.7% precision (95% CI 90.3-98.1%) and 67.3% recall (59.8-74.0%), ensuring extracted parameters are strictly factual. However, precision identifying absent parameters does not exceed 47.0%, establishing that the measured omission rate constitutes a loose upper bound on true literature incompleteness. Across three hardware configurations, complete GPU residency governs extraction quality more fundamentally than weight or cache precision, raising Matthews correlation from 0.414 to 0.530 at fixed quantisation and to 0.560 with an unquantised cache. A corpus-scale audit indicates only 19 (33.3%) of the 57 studies are reproducible in principle, reporting every method parameter needed to re-initialise the calculation. Driven to convergence, the workflow reproduces published lattice constants with a mean absolute relative error of 2.3% where the relaxed structure retains its prototype, establishing that lightweight open-weights models can reliably drive autonomous agentic workflows when bounded by deterministic code gates.
1 Introduction
This work addresses reproducibility, cost, and privacy problems in hosted-model agentic workflows by testing a fully local open-weights workflow for autonomous DFT auditing. It asks whether computational studies report enough methodological information for independent replication.
- Motivation: Hosted commercial models create reproducibility, cost, energy, and data-privacy vulnerabilities for computational materials workflows.Commercial endpoints can change or expire, metered inference makes iterative debugging expensive, and isolated networks may prohibit third-party data transmission.
- Related work: Smaller tool-augmented models can match or outperform larger unconstrained systems on selected chemistry and retrosynthesis tasks.Prior studies report a 4B retrosynthesis planner surpassing a 7B reasoning baseline and 7–8B chemistry agents matching datacentre-hosted equivalents.
- Research gap: The literature has not tested whether first-principles studies report sufficient methodological parameters for autonomous simulation recomputation across a corpus.Existing DFT reproducibility work assumes methodological parameters are completely and transparently reported.
- Approach: The study deploys a fully open-source local workflow using qwen3:4b, Ollama, and SIESTA, with agents restricted to named options and deterministic code gates.Every proposal is checked before reaching a calculation or reproducibility tally.
- Contributions: The contributions combine neurosymbolic extraction, input translation, and autonomous execution with a taxonomy of hardware- and prompt-related failure modes.The case study audits 57 first-principles studies of pentagonal two-dimensional materials and recomputes published lattice constants.
2 Methods
The methods implement a deterministic, locally executed pipeline that retrieves and validates full texts, extracts code-dependent parameters with grounded multi-pass inference, and drives guarded DFT recomputation. Reproducibility is defined by whether every parameter required to re-initialise the reported code is present, while inconclusive physical comparisons are withheld.
- System architecture: The four-phase pipeline retrieves documents, extracts parameters, translates claims into SIESTA inputs, and recomputes simulations to convergence.LLM agents operate within a workflow whose other stages are deterministic code steps.
- Hardware constraints: The constrained deployment uses qwen3:4b locally on a 4 GiB RTX 3050 laptop, with quantised weights and KV cache.A 15 GiB Tesla T4 is used to separate quantisation effects from uncontrolled GPU residency.
- Inference constraints: Generation uses greedy decoding, bounded outputs, disabled internal reasoning, dynamic contexts, retry rules, and GBNF grammar constraints derived from the target JSON schema.Documents exceeding the 32,768-token ceiling are recorded as failures.
- Neurosymbolic control: Agents propose named tool actions, while deterministic gates validate returned objects, bind arguments, restrict execution, and log interventions.The architecture separates agent proposals from the code-approved actions ultimately applied.
- Corpus construction: The corpus is built through OpenAlex retrieval, deterministic filtering, deduplication, full-text validation, and hash-based subsampling.Supplementary information is not fetched, and parameters reported only there are recorded as absent.
- Parameter extraction: A study is reproducible in principle only when it reports every code-dependent parameter required to re-initialise its calculation.The schema contains twenty parameters, with requirements differing between plane-wave and numerical-atomic-orbital codes.
- Recomputation and comparison: The recomputation stage reports raw physical deviations and withholds comparisons when calculations fail convergence or leave prototype symmetry.This avoids categorical verdicts when prototype starting geometries and independent basis sets confound methodological differences with reporting divergence.
3 Results
The audit finds that reproducibility varies with publication route and extraction reliability, while hardware residency and deterministic safeguards shape local workflow performance. Across 57 studies, only 19 meet the strict reproducibility criterion, and measured absences remain upper bounds because extraction can miss reported parameters.
- Retrieval yield and corpus composition: 10 of 17 (59 %) preprints state every required parameter, compared to 9 of 40 (22 %) publisher versions.The difference is 36 percentage points (95% CI 10–63 pp, Fisher’s exact test, p = 0.013).
- Local inference stability: 21 of 69 papers experienced total extraction collapse at least once across three passes; the grounded union recovered parameters for 4, while 17 failed in every pass.The union yielded 475 supported fields, recovering 35 over the best individual pass, but collision resolution removed nine fields from the naive union.
- Local inference stability: Full GPU residency produces a larger extraction effect than cache precision, raising recall and Matthews correlation coefficient at fixed q4_0 cache.Unquantising the cache trades presence recall for greater conservatism against ungrounded assertions.
- Extraction accuracy and reporting completeness: 95.7 % precision and 67.3 % recall characterize extraction against expert judgements, while absence precision does not exceed 47.0 %.The measured non-reporting rate combines genuine author omissions with extractor false negatives, so flagged absences are upper bounds on omission.
- Extraction accuracy and reporting completeness: The grounding gate removed 51 ungrounded values, preventing them from propagating into downstream simulations, while two evidence-free fields remained uncaught.Supporting sentences were accurately located in 93.3 % of cases, and 95.3 % of 169 extracted numeric values were correctly derived from quotes.
- Extraction accuracy and reporting completeness: Only 19 (33.3 %) of 57 papers are reproducible in principle in the main text, with a mean parameter reporting rate of 68.3 %.Omissions are concentrated in convergence controls, and 21.1 % of papers do not name the specific software package.
4 Discussion
The workflow’s high precision supports factual parameter assertions, but incomplete residency, limited source access, and missing structures constrain completeness and physical reproducibility. These limitations also expose campaign-level defects and geometry-dependent failure modes.
- Extraction and reporting completeness: 67.3% recall makes measured omission rates conservative upper bounds, because false negatives inflate apparent author omissions.The extractor achieved 95.7% precision with a 95% CI of 90.3–98.1% across nine parameters.
- Extraction and reporting completeness: Publisher supplementary files and separately indexed DOIs are excluded, so reported rates measure full-text presence rather than absolute parameter presence.Resolving this boundary requires dedicated per-publisher supplementary retrieval.
- Structural reproducibility: Structural reproducibility can fail despite methodological completeness when relaxed geometries are withheld and rebuilt prototypes determine the result.The resulting sensitivity can conflate methodological divergence with starting-geometry effects.
- Structural reproducibility: Threshold calibration lacked the six structures required for a defined tolerance, while basis sets, pseudopotentials, and starting geometries remained confounded.The calibration set contained only two of the six distinct structures required.
- Structural reproducibility: The carbon target became trapped in a different local minimum, expanding b/a to 1.186 and flattening the lattice to ∆z = 0.05 Å.By contrast, relaxed penta-PdSe2 retained pentagonal symmetry and buckling with ∆z = 1.63 Å.
- Campaign provenance: The retained logs record at least 509 extractions and approximately 4.7 million input tokens across ten campaigns before the reported corpus was produced.Configuration-stamped outputs exposed three verification records and three unreviewed critic-collapse verdicts.
5 Conclusions
The study concludes that a locally running Qwen3:4B workflow can support reproducibility auditing when deterministic constraints govern its assertions. Physical recomputations show that missing structural data, rather than omitted parameters alone, remains the primary barrier to reproducible discovery.
- 5 Conclusions: An open-weights qwen3:4b model on consumer hardware drove an end-to-end first-principles workflow under deterministic code gates.Assertions required verbatim source quotations and document-located magnitudes before informing calculations.
- 5 Conclusions: Successful parameter extraction does not guarantee structural reproducibility, because relaxed coordinates are rarely deposited and workflows rely on topological prototypes.Prototype-based initialization can produce conformational entrapment and confound methodological divergence with starting-geometry sensitivity.
- 5 Conclusions: Future verification workflows should retrieve publisher-specific supplementary structures and build community-driven reference geometry datasets.The framework is described as property-agnostic beyond density functional theory relaxations.
Data and code availability
The complete pipeline implementation and configuration files are available under an MIT license, alongside extracted records, labelled ground truth, and SIESTA inputs and outputs.
- Data and code availability: The repository provides source code, configuration files, agent-extracted records, hand-labelled ground truth, and all SIESTA input/output files.The retrieval module supports corpus regeneration from public endpoints through OpenAlex queries and repository fallbacks.
Conflicts of interest
The paper reports no conflicts of interest.
- Conflicts of interest: The authors declare no conflicts of interest.
CRediT authorship contribution statement
Shambhu Bhandari Sharma contributed across the paper’s conceptual, technical, analytical, validation, and writing activities.
- Shambhu Bhandari Sharma handled conceptualization, methodology, software, and formal analysis.
- Shambhu Bhandari Sharma conducted the investigation, data curation, and validation.
- Shambhu Bhandari Sharma contributed visualization and both original-draft and review-and-editing writing.