PaperlayerView exact revision on arXiv

Source-linked AI summary

HERA: Assessing an Autonomous Agent for Quantum Device Measurement

Zachary W. Gomez, Nathan Bittner, Jordan T. McCourt, Qingrui Cao, Sandesh S. Kalantre, Benjamin H. Alexander, Zhiren Zheng, Kenji Watanabe, Takashi Taniguchi, Aaron Sharpe

arXiv:2610.39074v1cond-mat.mes-hallcs.AI

TL;DR

HERA addresses the challenge of making autonomous experimental conclusions reliable when model behavior is stochastic and context-sensitive. It combines a lab-curated corpus with autonomous cryogenic transport measurement and evaluates it through blinded trials, ablation, offline replays, and a new-material test. HERA matched expert conclusions in the trial, while the corpus was associated with better-designed measurements and sounder reasoning across the reported tests.

  • Problem

    Autonomous experimental conclusions need scrutiny because LLM behavior is stochastic and context can steer conclusions, making controlled assessment important.

    Source: §I · p. 1

  • Method

    HERA combines a lab-curated corpus of protocols, software tools, and literature with an agent that conducts cryogenic transport measurement campaigns.

    Source: §I · p. 1

  • Results

    HERA matched the expert’s sealed conclusions on both trial devices, and tests found that the corpus supported better measurement design and more sound reasoning.

    Source: §I · p. 2, §VII · p. 8

  • Takeaways & Limitations

    With a task-designed knowledge base, an LLM agent can conduct a measurement campaign through to conclusions that withstand expert scrutiny.

    Source: §VII · p. 8

    This work focused on the measurement-campaign layer; the per-measurement optimization and research-program layers remain in development.

    Source: §VII · p. 9

Abstract

from arXiv · show

Large language model agents have begun to carry out laboratory measurements, but their scientific conclusions are useful only if they hold up under scrutiny. Demonstrating this requires controlled studies because models are stochastic and the context a model is given steers its conclusions. Here we introduce HERA, an autonomous agent designed to carry out cryogenic transport measurements of quantum devices. HERA draws on a knowledge base curated by lab members that holds measurement protocols, software tools, and literature references. We assessed HERA with a blinded trial against an expert on the stacking order of two multilayer graphene devices, an ablation of its knowledge base, and hundreds of offline replays in which models designed the trial's central measurement without running it. HERA's conclusions matched the expert's sealed conclusions on both tested devices. The ablation and the replay studies measured the impact of the knowledge base across different models, showing that with it every model designed better measurements and reasoned more soundly. To test how readily HERA extends to a new problem, we tasked it with finding and characterizing superconductivity in a different material system. This required a few hours of expert time to develop a new experimental protocol, after which HERA did both autonomously. These results show that, with a knowledge base designed for the task, language model agents are capable of sound experimental science.

I. INTRODUCTION

Quantum-device campaigns require targeted measurements across a high-dimensional parameter space, and conclusions build on earlier interpretations. HERA combines instrument-control tools with a lab-curated corpus and a reviewed session archive to conduct measurement campaigns.

I. INTRODUCTION

  • Targeted campaigns are needed because quantum-device measurements span a parameter space too high-dimensional to search exhaustively, and each interpretation builds on earlier ones.Many experiments across the sciences share this structure.Source: §I · p. 1
  • HERA is designed to keep devices and rigs safe, collect measurements that answer a question, and draw data-supported conclusions using lab-specific knowledge and instrument access.Its corpus is curated by lab members and contains protocols, software tools, and literature references.Source: §I · p. 1
  • Because measurement context can steer stochastic models’ conclusions, assessing an LLM-based system requires careful controls.Even repeated runs with the same context may produce different outputs.Source: §I · p. 1

II. HERA

  • The corpus provides recipes, tools, and references as resources rather than step-by-step execution instructions, with tools packaging validated analyses and defined inputs and outputs.Recipes connect protocols to examples, tools, and related recipes; the self-improvement loop adds resources for repeated or error-prone patterns.Source: §II · p. 2
  • Operators supply a campaign brief and machine-readable device and rig cards, while HERA chooses which recipes to consult and what data to acquire.The brief defines goals, boundaries, and possible starting points; conclusions use fixed confidence levels with evidence rules defined by the corpus.Source: §II · p. 2
  • HERA records measurement and analysis code in separate live notebooks, while session notes and a transcript join the notebooks and raw data in an archive.After a session, a review agent proposes corpus or prompt revisions for the operator to accept or reject.Source: §II · p. 2

III. A BLINDED PROSPECTIVE TRIAL

In a blinded prospective trial, HERA measured two multilayer graphene devices to determine their stacking order. It matched the expert’s sealed conclusion in both stacking order and confidence level.

III. A BLINDED PROSPECTIVE TRIAL

  • The expert’s conclusions were sealed before each session, and a protocol fixed what was withheld, the required conclusion format, and the comparison procedure.The protocol also required publication of every session.Source: §III · pp. 2, 4
  • HERA’s sequence used gate checks and fixed-density SdH spot checks to guide the ranges of a Landau fan, whose SdH frequencies distinguish stacking orders.For the trilayer, spot checks showed cleaner oscillations on the hole side with a lower onset field, guiding the fan’s measurement side.Source: §III · p. 4
  • The trilayer fan resolved a single frequency consistent with the higher ABA frequency, while the lower ABA frequency was unresolved rather than absent.The extracted carrier mass was close to the ABA expectation and far from the ABC expectation, so both discriminators pointed to ABA.Source: §III · p. 4
  • A finer fan extending to lower field resolved the lower ABA frequency but left HERA’s stacking-order conclusion and confidence unchanged.The operator requested this follow-up after HERA’s conclusion.Source: §III · p. 5
  • HERA matched the expert’s sealed stacking-order conclusion and confidence level on both trial devices, drawing both conclusions autonomously.The trilayer session had one operator intervention before its conclusion about a safety concern and one afterward for a finer fan; the tetralayer session had none.Source: §III · p. 5

IV. CORPUS × MODEL

The corpus ablation compared two models using either the full corpus or one retaining only software tools. Both models agreed with the expert using the full corpus; ablated sessions made claims their measurements did not substantiate.

IV. CORPUS × MODEL

  • With the full corpus, both models agreed with the expert, whereas one ablated-corpus session disagreed and the other’s conclusion lacked sufficient supporting data.The ablated corpus retained only software tools; under both conditions the models planned reasonable measurements.Source: §IV · p. 5
  • Opus 5 with the ablated corpus concluded ABCA from the false premise that Bernal tetralayer remains metallic near charge neutrality at zero displacement field.The ablated sessions were shortest at 3.9 and 4.1 active hours; Opus 5 with the full corpus stopped before the full suggested measurements and concluded ABAB at lower confidence.Source: §IV · p. 5
  • The ablated-corpus sessions had the lowest inference scores, while measurement decisions were graded high in all five tetralayer sessions.With the full corpus, every verdict and its evidence were graded correct except the Fable 5 repeat; both ablated verdicts rested on four evidence lines, two graded wrong.Source: §IV · p. 6
  • Under a second expert’s grades, every full-corpus session still outscored both ablated-corpus sessions, despite disagreement on 51 of 145 jointly scored items.The experts gave the same grade on 65% of items, with quadratic-weighted Cohen’s κ = 0.38; about 80% of disagreements were single-step changes.Source: §IV · p. 6
  • Both ablated-corpus sessions were graded as needing re-measurement or failing to answer the brief, while two of three full-corpus sessions answered it as is.The remaining full-corpus session answered after re-analysis; the expert’s most common criticism across conditions was insufficiently high-resolution data.Source: §IV · p. 6
  • Every tetralayer session initially misread resistance peaks near charge neutrality, a feature no corpus recipe covers, but full-corpus sessions kept the peaks out of their stacking conclusions.Opus 5 with the ablated corpus treated one peak as charge neutrality and inferred a gap that would rule out ABAB; the Fable 5 repeat carried the two-region interpretation into its verdict.Source: §IV · p. 6

V. OFFLINE REPLAY

Offline replays compared Landau fans designed with HERA’s full and ablated corpora, allowing six models’ measurement designs to be scored without running them. Every model designed higher-scoring fans with the full corpus, which also reduced planned acquisition time.

V. OFFLINE REPLAY

  • In 300 replays, six models each designed up to twenty fans per corpus condition; 239 designs were scoreable, while most unscored replays designed another spot check.The full-corpus recipe emphasizes spot checks before fan design.Source: §V · p. 6
  • Every model’s fans scored higher on P(call) with the full corpus, and the pooled gain was significant (permutation test, p = 0.0004 after Holm correction).The replay compared fans under an eight-hour cutoff, counting longer designs as zero.Source: §V · p. 7, Figure 4 · p. 7
  • Without the cutoff, the full corpus raised mean P(call), lowered median planned acquisition time from 5.5 h to 4.4 h, and had Pareto dominance of 0.40.The proportion of paired fans better on both P(call) and time was 45% for the full corpus and 5% for the ablated corpus.Source: §V · p. 7, Figure 4 · p. 7
  • With the full corpus, every model had a higher mean sampling rate, all but GPT-6 Astra had higher mean frequency coverage, and every model’s designs were less dispersed.The four Claude models converged on nearly the same design, while the two GPT models landed elsewhere.Source: §V · p. 7, Figure 4 · p. 7

VI. DOMAIN TRANSFER

HERA was tested on superconductivity in WSe2-proximitized bilayer graphene using its existing harness, agents, and corpus, with a new device card, rig card, and recipe. After recipe corrections, HERA autonomously found and characterized the superconductor.

VI. DOMAIN TRANSFER

  • The transfer reused HERA’s harness, agents, and existing corpus, requiring a device card, rig card, and superconductivity recipe; an expert composed the documents in a few hours.The recipe encoded the expert’s evidence standard and suggested measurement order, grounded in Refs. [19].Source: §VI · pp. 7–8
  • The test system was WSe2-proximitized Bernal bilayer graphene, whose superconductivity had previously been confirmed by an expert in a narrow region of the n–D plane.HERA ran six sessions over six days.Source: §VI · p. 7
  • The second session surveyed only to D/ϵ0 = −0.60 V/nm and incorrectly concluded the device did not superconduct; an operator message prompted a sweep at higher |D| that identified it.The recipe was corrected to recommend covering the whole accessible gate range before a negative conclusion.Source: §VI · p. 8
  • With two recipe corrections, HERA autonomously found and characterized the superconductor, including the temperature dependence of critical current; extracted quantities were within about a factor of two of an operator-in-the-loop reference.The spread reflected where sessions chose to measure within the superconducting region and how they extracted each quantity.Source: §VI · p. 8

VII. DISCUSSION

The discussion argues that HERA’s conclusions depend on both measurement tools and evidence rules, and assesses the agent against fixed references. It identifies limits of replay and the work’s focus on the campaign layer.

VII. DISCUSSION

  • With software tools alone, both models took too little data to substantiate their stacking claims; evidence rules helped prevent unsupported conclusions even when recipes missed a feature.Every tetralayer session initially misread resistance peaks near charge neutrality, but with the full corpus no session based its stacking conclusion on those peaks.Source: §VII · p. 8
  • A mistake in the corpus can affect every session that reads it, so changes are human-reviewed and version-controlled; replay tests decisions on acquired data, not recipes end to end.A virtual lab with simulated instruments and a device forward model could test recipes from start to finish without cryostat time.Source: §VII · pp. 8–9
  • This work focused on the measurement-campaign layer; real-time measurement optimization and research-program hypothesis generation remain in development.Together, those layers would form an end-to-end discovery loop.Source: §VII · p. 9

devices.

  • K.W. and T.T. supplied boron nitride crystals, Z.W.G. and N.B. built the offline benchmark, and A.S. graded sessions and supervised the project.Z.W.G. analyzed the data; Z.W.G. and A.S. wrote the manuscript with input from all authors.Source: devices · p. 9

COMPETING INTERESTS

  • The work was supported by an Anthropic AI for Science Program award; Anthropic had no role in study design, data collection, analysis, or publication decisions.N.B. was employed by Amazon Web Services, which has a commercial relationship with Anthropic; the work was conducted independently of Amazon and without its support.Source: COMPETING INTERESTS · p. 9

VIII. METHODS

HERA sessions combine operator-supplied device, rig, and goal information with agent-selected measurements and analysis. Autonomy is safety-gated, while the versioned corpus supplies protocols, tools, and evidence guidance.

Implementation

  • The operator supplies a brief, device card, and rig card, while the agent chooses which recipes to consult and what data to acquire, then records measurement code and reasoning in live notebooks.The agent analyzes and states conclusions in a separate analysis notebook.Source: Implementation · p. 9

Autonomy and operation

  • E0 requires the operator to execute measurement cells; E1 lets the agent execute them only after safety checks pass, and autonomy can be withdrawn if the agent raises a concern.An intervention is any operator message after autonomy is granted, except an unchanged resend after a provider outage; active time excludes standby.Source: Autonomy and operation · p. 10

Corpus

  • The version-controlled corpus held 29 recipes, worked examples, papers, and forward and Raman models; its rules specify conditions, source types, and confidence levels for claims.The ablated corpus retained software tools but removed the forward and Raman models; trial and corpus-by-model sessions used frozen or ablated versions.Source: Corpus · pp. 10–11

Review and revision

After sessions, a review agent gathers findings, checks prior fixes, and recommends corpus changes for operator adjudication. The section also describes device fabrication and the gate-voltage relationships used to define carrier density and displacement field.

Review and revision

  • The review agent checks session findings against prior revisions and deferred findings, recommends whether to revise, defer, record, or drop each issue, and proposes edits for operator adjudication.Accepted changes become version-controlled commits and fix-log entries; periodic cleanup removes text without adding any.Source: Review and revision · p. 11

Devices and fabrication

  • Devices were hBN-encapsulated graphene stacks two to four layers thick, with graphite top and bottom gates; fabrication and stacking identification methods varied across devices.Devices 1–3 were stacking-order trial or development devices, while device 4 was a Bernal bilayer graphene/WSe2 device for the superconductivity search.Source: Devices and fabrication · p. 11
  • Carrier density and displacement field are calculated from top- and bottom-gate voltages and capacitances; the safe gate-voltage ranges bound a parallelogram in the n–D plane.The equations use the top and bottom gate capacitances per unit area.Source: Devices and fabrication · p. 11

Rigs

The graphene and WSe2-proximitized devices were measured in different cryostats with distinct base temperatures and filtering. The blinded-trial protocol sealed expert conclusions before sessions and compared them with HERA’s committed conclusions.

Rigs

  • The graphene devices were measured in a variable-temperature insert at about 1.2 K without line filtering, while the WSe2-proximitized device used a filtered dilution refrigerator at about 30 mK.Both setups used lock-in amplifiers for four-terminal resistance measurements.Source: Rigs · pp. 11–12

Blinded trial protocol

  • The trial protocol was frozen before sessions, required publication of every eligible device in measurement order, and accepted a reasoned “cannot tell” as a valid outcome.A confident conclusion without feature-level evidence violated the protocol.Source: Blinded trial protocol · p. 12
  • Experts sealed their conclusions before sessions; HERA received no Raman or prior device measurements, and its committed conclusions were compared with the expert’s for stacking order and confidence-level agreement.The protocol reports agreement, not accuracy.Source: Blinded trial protocol · p. 12

Corpus × model protocol

The corpus × model study compared full and ablated resources across models, while grading and replay protocols specified how reasoning and measurement designs would be assessed.

Corpus × model protocol

  • Two models each ran once with the full corpus and once with an ablated corpus that retained software tools, cards, and safety limits.The ablation removed recipes, worked examples, forward models, papers, the style guide, and method documents; the ablated sessions used a separate brief with the same goal and limits.Source: Corpus × model protocol · p. 12

Grading

  • Claims were extracted, split into checkable items, and locked before grading; item scores were averaged into claim and session grades.Correct, partly correct, and wrong items scored 1, 0.5, and 0, respectively; unverifiable items were excluded, and verdict claims received double weight in the overall claim score.Source: Grading · pp. 12–13
  • A second expert graded a fixed subset, while the first expert also assigned whole-session grades for answering the brief and required future supervision.The second expert graded all verdict claims and 30% of inference and decision claims; whole-session grading was not blind to device identity.Source: Grading · p. 13

Replay benchmark

  • Replay agents started from an archived state before fan design, and only completed fan designs were graded against expected tetralayer SdH frequencies.Each model designed twenty fans per corpus when possible; the grader checked sampling and record-length requirements for resolving the relevant frequencies.Source: Replay benchmark · p. 13
  • The replay benchmark scored designed Landau fans for their ability to distinguish stacking orders using pre-registered metrics and acquisition-time constraints.P(call) combined the ability to resolve the light Bernal frequency and separate the rhombohedral frequency from the heavy Bernal frequency; a fan taking over eight hours scored zero.Source: Replay benchmark · pp. 13–14
  • At the twelve-hour sensitivity cutoff, Fable 5’s gain was −0.02, while every other model’s gain remained positive at all tested cutoffs.Eleven of Fable 5’s twenty full-corpus fans took 9.4 to 20.4 hours, compared with 7 to 9 hours for its best ablated-corpus fans.Source: Replay benchmark · p. 14

Cost

The section describes experimental costs and records, then explains how Landau fans and related measurements support stacking-order conclusions. It also reports HERA’s frequency and mass evidence and a contrasting tetralayer interpretation.

Cost

  • Token costs were calculated by token class using list prices, with cached input priced below uncached input and cache writes priced above it.One-hour cache writing cost twice the input price; uncached input and output were priced separately by model.Source: Cost · p. 14

S2. MODELS AND VERSIONS

  • The section identifies TABLE S1 as the record of models, harnesses, corpora, and autonomy modes used across experiments.The caption notes that some transfer-session model records were incomplete and that a development false start was omitted.Source: Table S1 · p. 19

S3. DETERMINING STACKING ORDER FROM A LANDAU FAN

  • At the tested trilayer density, the light ABA frequency distinguishes ABA from ABC more clearly than the heavier ABA frequency, which lies near the ABC frequency.The model places the ABA frequencies at 2.3 and 15.7 T and the ABC frequency at 18.0 T; density calibration is uncertain by about 10%.Source: S3. DETERMINING STACKING ORDER FROM A… · p. 20
  • The stacking-order recipe uses frequency counting as the primary discriminator, treats resolved level crossings as complementary evidence, and permits a reasoned “cannot tell.”It recommends short SdH spot checks and checks for spectral artifacts before counting frequencies.Source: S3. DETERMINING STACKING ORDER FROM A… · p. 20
  • A Landau fan’s frequency resolution depends on its 1/B range, while its step size must sample the highest frequency to avoid aliasing.The stated requirements are δF = 1/(1/Bmin − 1/Bmax) and ∆(1/B) < 1/(2Fmax); HERA’s trilayer fan covered 1.6 to 5 T, spanning about one period of the light ABA frequency.Source: S3. DETERMINING STACKING ORDER FROM A… · p. 20
  • HERA’s two spot checks found stable hole-side frequencies and an onset near 1.7 T, so it planned the fan on the hole side.The electron-side check had an onset near 2.2 T and a low-frequency region sensitive to background choice.Source: S3. DETERMINING STACKING ORDER FROM A… · p. 20
  • HERA’s fitted dominant frequency had a cyclotron mass of 0.049 ± 0.002 me, consistent within model uncertainty with the heavier ABA band and used to exclude ABC.The model gives 0.060 me for the heavier ABA band and 0.11 me for ABC; HERA did not measure the light ABA-band mass.Source: S3. DETERMINING STACKING ORDER FROM A… · p. 21
  • In the tetralayer sessions, all five initially read the zero-field resistance peaks as differing channel doping or gating, but full-corpus sessions did not base their conclusions on that interpretation.The ablated Opus 5 session treated a ten-fold resistance rise under displacement field as evidence against Bernal stacking; the expert graded that reasoning wrong.Source: S3. DETERMINING STACKING ORDER FROM A… · p. 21

S4. THE SIX LIVE STACKING-ORDER SESSIONS

The section’s table summarizes six live stacking-order sessions, and its figure shows the expert’s unit-by-unit rulings across five tetralayer sessions.

S4. THE SIX LIVE STACKING-ORDER SESSIONS

  • TABLE S2 compiles each live session’s stacking conclusion, expert agreement, interventions, timing, costs, claim scores, and expert letter grades.The table also distinguishes measurements before and after HERA’s conclusion and reports standby and post-conclusion sweep time.Source: Table S2 · p. 22
  • FIG. S3 arranges graded notebook units by session and reading order, coloring each unit by the expert’s ruling and outlining units containing a graded inference or verdict.Context units are omitted, and the five-session order follows Fig. 3.Source: Figure S3 · p. 23

S5. OPERATOR INTERVENTIONS

The section records operator involvement during live sessions and compares two experts’ grading of a claim subset. Their agreement was conditional on which items both experts judged verifiable.

S5. OPERATOR INTERVENTIONS

  • On device 1, the operator ruled HERA’s safety stop acceptable, and a finer fan requested afterward resolved the feature HERA had flagged as unresolved.The session involved 16.6 hours on standby, including 11.4 hours awaiting the operator’s ruling.Source: S5. OPERATOR INTERVENTIONS · p. 24
  • On device 2, full-corpus Fable 5 sessions had no interventions, while ablated Opus 5 required one before its conclusion to approve a lock-in setting change.After conclusions, the operator requested a finer fan from full-corpus Opus 5 and sent close-out messages in both ablated sessions.Source: S5. OPERATOR INTERVENTIONS · p. 24

S6. SECOND-EXPERT GRADING OF THE CLAIM SUBSET

  • The two experts differed in their overall agreement on subset scores, with exact agreement of 0.453 and weighted κ of 0.508 across 53 claims.At the item level, exact agreement was 0.648, below the sealed prediction of at least 0.70.Source: Table S3 · p. 25, S6. SECOND-EXPERT GRADING OF THE CLAIM… · p. 25
  • The second expert’s grading preserved the pattern that every full-corpus session outscored both ablated sessions, despite the experts’ item-level disagreements.The second expert’s margin between Fable 5 repeat with the full corpus and Fable 5 with the ablated corpus was 0.002.Source: Table S4 · p. 25, S6. SECOND-EXPERT GRADING OF THE CLAIM… · p. 25
  • The experts disagreed on 51 of 145 jointly ruled items, with 15 disagreements coming from the Fable 5 ablated-corpus session.Their grading notes differed in whether to penalize poor data quality before assessing reasoning or to grade reasoning while noting caveats.Source: S6. SECOND-EXPERT GRADING OF THE CLAIM… · p. 25
  • The second expert ruled 51 items unverifiable and the first ruled 24, leaving the reported agreement conditional on both experts having ruled.Only eight unverifiable items were common to both experts.Source: S6. SECOND-EXPERT GRADING OF THE CLAIM… · p. 25

S7. SUPERCONDUCTIVITY IN THE PROXIMITIZED BILAYER GRAPHENE DEVICE

The superconductivity search combined transport measurements and Ginzburg–Landau analysis, while the merged sections also report an offline-replay cutoff comparison and a development campaign on device 3.

S7. SUPERCONDUCTIVITY IN THE PROXIMITIZED BILAYER GRAPHENE DEVICE

  • Sessions 2 and 4 observed a single central lobe, with critical current largest at effective zero field and closing by 2.5–3 mT; absent oscillations did not count against superconductivity.The search treated oscillations as conditional on disorder forming a weak link.Source: S7. SUPERCONDUCTIVITY IN THE… · p. 26
  • The 50% transition criterion yielded Bc2(0) = 3.0 mT and ξGL = 329 ± 9 nm.The analysis extracted Tc where resistance reached 50% of its normal-state value.Source: S7. SUPERCONDUCTIVITY IN THE… · p. 28, Figure S5 · p. 28
  • HERA surveyed the gate window and selected operating points before characterizing resistance and bias/field response at the chosen superconducting location.The survey figure overlays each session on the reference resistance map and marks the accessible gate range.Source: Figure S4 · p. 27, Figure S6 · p. 29

S8. OFFLINE REPLAY: GAIN AT EACH CUTOFF

  • At seven hours, Fable 5’s full-corpus P(call) gain was +0.19, versus −0.13 at nine hours; eight hours was the primary cutoff.The other cutoffs were pre-registered sensitivity analyses.Source: Table S6 · p. 30

S9. DEVELOPMENT OF THE STACKING INVESTIGATION

  • HERA ran nine sessions on device 3; the last three unchanged-brief runs varied between ABA and ABC because the light ABA band was below resolution.The figure shows four runs; run 6 was a false start and is not counted.Source: Figure S7 · p. 31
Loading 2610.39074v1…