Source-linked AI summary
Collaborative AI agents for autonomous experimental research with solid-state quantum spins
El Mustapha Mansouri, Eikichi Kimura, Masato Koga, Keigo Arai
TL;DR
Collaborative AI agents can develop theoretical claims and test them experimentally, but reliable conclusions require scrutiny of both predictions and experimental controls. In a diamond-spin calibration study, the agents derived and computationally checked an exact calibration law, then designed a discriminating experiment whose revised measurements supported its predictions.
Problem
A demanding test for AI research agents is whether they can develop a theoretical claim and design experiments that test it, rather than draw conclusions unsupported by evidence.
Source: §6 · p. 8Method
Results
Takeaways & Limitations
The investigation shows how an AI laboratory can identify a precise information requirement, turn it into a measurable prediction and assess the evidence for it.
Source: §6 · p. 9Failures of global-phase and reference-pulse controls were associated with signal changes sampled at different times by the two programs, prompting acquisition amendments.
Source: §7.8 · p. 13
Abstract
from arXiv · showhide
An autonomous research laboratory could conduct an entire study, from formulating questions to preparing manuscripts. Artificial intelligence (AI) offers one route towards this goal. Recent AI systems take on several research roles, but may draw conclusions that the evidence does not support. Here, we present an autonomous research laboratory in which agents based on large language models assess one another's work and can stop or redirect investigations. They can generate and reject research questions, require separate theoretical checks and conduct experiments designed to challenge their claims. We demonstrate these capabilities by investigating ensembles of quantum spins in diamond. The agents derived the minimum number of calibration settings needed to predict responses to a defined family of microwave pulse sequences. They checked the result computationally and designed experiments to distinguish competing predictions. An initial experiment agreed with the theoretical predictions, but repeated reference measurements showed that the signal was drifting during data collection. The agents therefore treated the result as inconclusive and revised the measurement protocol without relaxing the acceptance criteria. The revised experiment met those criteria and supported the theoretical predictions. This study demonstrates that collaborating AI agents can select research questions, derive theoretical results and test their predictions experimentally.
1 Collaborative AI laboratory for scientific discovery
The laboratory distributes research across specialized agents whose outputs are independently evaluated before the investigation advances. Its workflow connects question selection, theory, computation and experimentation, with human-configured safety boundaries.
1 Collaborative AI laboratory for scientific discovery
- Agents can reject collaborators’ work or request revisions, with mathematical claims independently checked and experiment designs reconsidered when they cannot distinguish alternatives.Experimental predictions and acceptance criteria are fixed before target data are analysed.Source: §1 · pp. 2–3
- Six roles divide proposal generation, oversight, computation, experimentation, coordination and independent manuscript review.Proposers generate questions; the PI selects and develops one, while the computational researcher checks results and assumptions, and the experimentalist performs measurements.Source: §1 · p. 2
- Human researchers assembled the apparatus and set preset operating limits; safety-agent approval was required before experimentation.Source: §1 · p. 3, Figure 1 · p. 3
2 Critical evaluation and selection of research questions
The PI evaluated proposals for novelty, logic, significance and feasibility, rejecting batches and using criticism to direct further proposals. Selection followed revision of a rejected idea rather than acceptance of an initial proposal.
2 Critical evaluation and selection of research questions
- Proposals were assessed for novelty, logical validity, significance and experimental feasibility, and the PI could reject an entire batch with written criticism.Proposers worked independently from one another.Source: §2 · pp. 3–4
- The PI rejected 75 proposals over approximately 30 hours, and the manager excluded recurring rejected ideas from later instructions, after which proposers explored seven new directions.Those ideas had appeared in 52 of the 56 earlier fresh proposals; nine later proposals explored seven previously unrepresented directions.Source: §2 · p. 4
- A previously rejected idea became the seed for a revised question about the minimum calibrations needed to predict arbitrary phase-controlled pulse-sequence responses.The revision retained full dependence on pulse sequences rather than relying on a leading-order approximation.Source: §2 · p. 4, Figure 2 · p. 4
3 Discovery and verification of an ensemble-calibration law
The PI derived an exact calibration requirement for predicting ensemble responses to phase-controlled pulse sequences, and a computational agent independently checked the result and tested robustness to hidden phase rotations.
3 Discovery and verification of an ensemble-calibration law
- L^2 + 1 independent calibration measurements are necessary and sufficient for predicting responses to sequences of at most L pulses under persistent spin errors.For L = 3, this reduces the proposed upper bound of 91 calibrations to 10; the proof allows an unknown common fluorescence background and contrast.Source: §3 · p. 5
- The law predicts a family of responses without reconstructing the unknown spin-error distribution, provided calibration settings have linearly independent responses.Repeated acquisitions determine the precision of the measured means, not the number of distinct calibration settings.Source: §3 · p. 5
- Independent computation reproduced symbolic checks and response-space dimensions, while 252,620 certificates tested ambiguity and prediction error under hidden phase rotations.For the specified six-pulse test, certified prediction error remained at photon-noise level for hidden rotations of approximately 1.2° or less.Source: §3 · p. 5
4 Collaborative design of a discriminating experiment
The agents designed an experiment to distinguish the calibration law from interpolation methods, rejecting a design whose targets could already be predicted by Fourier interpolation. They revised the calibration set and fixed predictions and acceptance criteria before target analysis.
4 Collaborative design of a discriminating experiment
- The initial nine-point grid of three-pulse calibrations could predict the withheld targets through Fourier interpolation, so agreement would not distinguish the law from interpolation.Source: §4 · p. 5, Figure 4 · p. 7
- The revised design combined five three-pulse calibrations with five shorter sequences to predict four withheld three-pulse targets that interpolation could not determine.The shorter sequences supplied information missing from the five three-pulse settings.Source: §4 · p. 6, Figure 3 · p. 6
- Predictions and acceptance criteria were fixed before target analysis, and independent controls defined outcomes as supporting, contradicting or inconclusive.An acquisition that failed a control was classified as inconclusive, even if predictions agreed.Source: §4 · p. 6
5 Rejecting an apparently successful run and establishing experimental support
The final acquisition passed all accuracy criteria and experimental controls, supporting the calibration law’s predictions for four withheld responses. An earlier run met the accuracy criteria but was rejected as inconclusive after controls exposed signal drift.
Initial acquisition
- The first run met all ten accuracy criteria but failed global-phase and reference-pulse controls, so the PI classified it as inconclusive.The signal rose by 7.6% over the morning and stepped another 2.9% when room lights came on, creating offsets between programs sampled at different times.Source: §5 · p. 7, Figure 5 · p. 8
Revised protocol
- The agents revised acquisition without relaxing acceptance criteria, and a pilot using phase switching within one program passed the phase controls.The PI specified a new six-hour acquisition with randomized sequence ordering and additional null-pair diagnostics.Source: §5 · p. 7
Final acquisition
- (40.0 ± 0.4) × 10−3 weighted counts per shot matched the law’s (39.7 ± 1.3) × 10−3 prediction; both interpolation alternatives predicted zero.The final acquisition yielded 21,600 valid one-second records and passed all ten accuracy criteria and experimental controls; predictions were fixed before target analysis.Source: §5 · pp. 7–8, Figure 5 · p. 8
6 Discussion
The study presents a calibration law whose exact information requirement was experimentally tested against interpolation alternatives. It also identifies limits to greater autonomy in the agent architecture and experimental infrastructure.
Scientific judgement
- The agents distinguished predictive agreement from sufficient scientific support by rejecting a control-failing run and revising the protocol without relaxing criteria.Critical evaluation shaped the research question, strengthened the theory and determined which experimental evidence could support a conclusion.Source: §6 · p. 8
Calibration law
- For sequences of at most L pulses under persistent pulse errors, L2+1 measurements are necessary and sufficient to predict the ensemble response.Necessity was established using physically admissible ensembles that agree on too few calibrations yet differ on unmeasured sequences.Source: §6 · p. 8
Experimental test
- Measurements supported the law’s predictions for four withheld targets within predefined tolerances, distinguishing them from two specified interpolation methods.The result connects the theoretical information requirement to measurable predictions and an experimental test.Source: §6 · pp. 8–9
Platform prospects
- NV centres offer optical spin initialization and readout with programmable microwave control, and similar approaches may apply to other quantum platforms.Their sensitivity to control imperfections and environmental fluctuations provides demanding tests of quantum-control theories.Source: §6 · p. 9
Limits and extensions
- Greater autonomy will require improvements in agent architecture and experimental infrastructure, including more reliable diagnostics, coordination and safety mechanisms.These needs grow as agents gain additional experimental degrees of freedom.Source: §6 · p. 9
Future testbeds
- Broader autonomy also depends on testbeds that make predictions experimentally accessible and competing explanations distinguishable without human involvement.The authors describe a possible future shift in human roles toward identifying consequential questions and critically assessing automated discoveries.Source: §6 · p. 9
7 Methods
The methods describe the agent roles and human contributions, information exchange and review process, and adaptive selection of research proposals.
7.1 Agent implementation and human contributions
- The laboratory used model conversations and coding-agent sessions for distinct roles: the PI used GPT-6 Pro, proposal generation GPT-5.6 extra high, and computation GPT-6 Astra at maximum reasoning.The experimentalist used Claude Fable 5 through Claude Code, while the manager used Claude Opus 5; human researchers assembled the NV apparatus and configured preset safety limits.Source: §7.1 · p. 9
7.2 Context allocation, document exchange and review
- Agents exchanged written documents relayed by the manager, while independent final reviewers saw the manuscript without preceding exchanges, code or raw data.The experimentalist kept local notes and logs; raw photon counts remained on the instrument computer.Source: §7.2 · p. 10
7.3 Proposal generation and adaptive selection
- Proposers generated physics-essential, testable questions; the PI screened proposals and used criticism and accumulated exclusions to guide revisions.The PI could reject a batch, and could select none only when no candidate was novel, feasible or impactful.Source: §7.3 · pp. 10–11
7.4 Spin-response model and calibration construction
The model considers fixed-duration, fixed-amplitude, fixed-shape pulses with independently chosen phases, persistent spin-specific pulse actions, and an unknown ensemble distribution. A basis of calibration responses spans the response space for sequences up to L pulses, allowing target predictions from measured calibration means with a stated photon-noise error bound.
7.4 Spin-response model and calibration construction
- Each spin retains the same pulse action throughout a sequence, while the ensemble is represented by an unknown probability distribution over an admissible domain.Pulse duration, amplitude, and shape are fixed; pulse phases are independently programmable.Source: §7.4 · p. 11
- dL = L^2 + 1 linearly independent calibration responses span the response space for sequences of at most L pulses.The construction uses five phases, 2πk/5, to resolve phase Fourier components.Source: §7.4 · p. 11
- Target predictions use real coefficients from expansion in the calibration basis and the measured mean responses for those calibration sequences.dL counts distinct calibration settings; repeated acquisitions determine their precision.Source: §7.4 · p. 11
- M dL/(N − dL + 1) bounds the worst-case mean-squared prediction error under the stated Poisson-count model and determinant-maximizing acquisition design.The analysis combines at least dL observations with least-squares estimation; acquisition means are bounded by M.Source: §7.4 · p. 11
7.5 Independent computational verification and robustness analysis
The computational verification reconstructed the response-space structure and used certified numerical tests to examine robustness to hidden rotations. The experimental design compared the calibration law with interpolation alternatives using locked predictions, predefined criteria, and reviewed feasibility.
7.5 Independent computational verification and robustness analysis
- For L = 5 and L = 6, the computational researcher independently reconstructed response-space coordinates after reproducing the symbolic checks.The robustness test examined whether hidden rotations between pulses could evade calibration while changing target predictions.Source: §7.5 · p. 11
- 252,620 exact certificates bounded prediction error and calibration ambiguity across specified hidden-rotation cases using rational arithmetic and Bernstein-coefficient checks.The tests covered every target, three rotation positions, the unperturbed case, and twelve evaluated rotation strengths; a Berry–Esseen bound tightened ambiguity bounds.Source: §7.5 · p. 12
7.6 Experimental design and prediction locking
- The final design used ten calibration sequences spanning the L = 3 response space to predict four withheld three-pulse targets, with interpolation alternatives and a physical model specified for comparison.A digital twin evaluated the decision rule, and the packet underwent separate review before execution.Source: §7.6 · pp. 12–13
- The PI fixed the transfer matrix, contrast definition, tolerances, controls, and outcome criteria before acquisition, and the protocol required predictions to be locked before target analysis.The protocol did not relax its prescribed thresholds.Source: §7.6 · p. 13
7.7 NV apparatus and fluorescence readout
The laboratory combined room-temperature NV-spin fluorescence measurements with phase-controlled microwave pulses. After early control failures associated with signal changes during acquisition, the protocol was amended while retaining its original decision criteria.
7.7 NV apparatus and fluorescence readout
- At room temperature, measurements on a single-crystal diamond with approximately 4.5 ppm NV centres used a confocal microscope; the detected rate was typically 2.8–3.1 × 10^6 counts s−1.The setup used gated continuous-wave 532 nm excitation, a 100× objective, and a single-photon counter.Source: §7.7 · p. 13
- An FPGA synchronized optical and microwave switching, while calibrated in-phase and quadrature waveforms controlled microwave phase and frequency offset.Final calibration and target pulses lasted 64 ns, with 152 ns between pulses; weighted counts from three readout windows formed each shot's signal.Source: §7.7 · p. 13
7.8 Acquisition protocol and amendments
- The first acquisition alternated two controller programs across 36 ten-minute sub-runs, but signal changes sampled at different times by the programs led to failures of the global-phase and reference-pulse controls.The first main acquisition was therefore inconclusive, prompting acquisition amendments.Source: §7.8 · p. 13
- The amended protocol used within-program phase toggling, randomized sequence orders, and 720 thirty-second blocks while retaining the original decision criteria.After a reference-diagnostic calculation error was corrected, the final acquisition completed six hours of integration with 21,600 valid one-second records.Source: §7.8 · p. 14
7.9 Statistical analysis and outcome classification
The statistical analysis tested predefined accuracy margins for the contrast and withheld-target residuals, propagated shared calibration uncertainty, and assessed experimental controls. Fixed rules classified outcomes as support, contradiction, or inconclusive, with separate later diagnostics.
7.9 Statistical analysis and outcome classification
- Positive margins required agreement within 20% of P for the contrast and residual magnitudes below 30% of P for each of four withheld targets.The ten margins comprised two contrast bounds and upper and lower bounds for each target residual.Source: §7.9 · p. 14
- Uncertainty propagation included calibration-derived prediction uncertainty and covariance between P and each residual; variance validity failed when F* exceeded 1.25.Poisson-based variance contributions used an excess-variance factor with a floor of 1.1.Source: §7.9 · p. 14
- Global-phase invariance, reference-pulse phase invariance, and null-signal consistency were tested using prespecified controls with rejection thresholds corresponding to a tail probability of 10−4.Later diagnostics of residual consistency and acquisition-order dependence were separate from the fixed outcome rule.Source: §7.9 · p. 14
- Support required valid controls and variance, positive one-sided 99% lower bounds for all ten margins, and positive S at p ≤ 0.001; failures of validity or other unmet rules were inconclusive.Contradiction required valid controls and variance, P more than 3.09 standard errors above zero, and at least one margin with a negative one-sided upper bound at tail probability 10−4.Source: §7.9 · p. 14
Competing Interests
The disclosures report no competing interests, assign responsibility for scientific claims and data integrity to the human authors, make data and code available on request, and release Auctopus as open source.
Competing Interests
- The authors declare no competing interests.Source: Competing Interests · p. 15
AI Use Disclosure
- Human authors take full responsibility for scientific claims, data integrity, and the final manuscript despite agents' broad research role.Source: AI Use Disclosure · p. 15
Data / Code Availability
- Supporting experimental data and analysis code are available from the corresponding author upon reasonable request.Source: Data / Code Availability · p. 15
Open-Source Release
- Auctopus is released under the MIT licence and runs the workflow from proposal generation through laboratory measurement and review.The release includes adaptable templates, but not the prompts used in this study.Source: Open-Source Release · p. 15