Source-linked AI summary
LLM-IDEA: Identifiability-Driven Experimental Agent for Autonomous Discovery of Mechanistic World Models
Surya Shetty, Ulisses Braga-Neto
TL;DR
Agents can plateau because of limited capability or because their data cannot identify the target, a distinction that matters when deciding whether more search can help. LLM-IDEA couples a language-model proposer with a deterministic identifiability engine that diagnoses confounded directions and evaluates candidate experiments. Across the reported validations, it identifies systems recoverable under existing protocols and flags cases where the protocol cannot resolve the target.
Problem
At a plateau, an agent may be unable to distinguish limited capability from parameters that the observation protocol cannot identify.
Source: §1 · pp. 1–2Method
LLM-IDEA couples a language-model proposer to a deterministic engine that diagnoses sensitivity rank, names confounded directions, ranks candidate designs, and certifies exhaustion within a design class.
Source: §1 · p. 2, §3.2 · p. 5, §3.3 · p. 5Results
On ODEBench, 60 of 62 systems with free constants were identifiable at round 0; on DiscoverPhysics, accurate-trajectory models failed the explanation grade in 15 of 15 unidentifiable worlds, versus 5 of 9 in identifiable worlds.
Source: §7 · p. 26Takeaways & Limitations
The three-way verdict distinguishes when to keep searching, run a better experiment of the same kind, or change the design class.
Source: §1 · p. 2The certificate depends on the declared design class, and local diagnostics can miss a second discrete solution elsewhere.
Source: §6 · pp. 25–26
Abstract
from arXiv · showhide
Large language model agents are being increasingly deployed as autonomous scientists, designing experiments and inferring mechanistic world models with minimal human oversight. Yet identifiability is often overlooked: when a plateau is reached, the agent needs to know whether it is not yet capable enough or the model simply is not identifiable from the data, in which case no amount of further experimentation of the same kind can help. We propose the Identifiability-Driven Experimental Agent (LLM-IDEA) for closed-loop discovery with an identifiability engine that returns a three-way plateau verdict: capability limit, resolvable within the design class, or certified exhausted. On ODEBench, 60 of the 62 systems with free constants are identifiable at round 0; the RC circuit is certified exhausted for every experiment that protocol can run, and a harvesting model is resolvable by one added initial condition. The identifiability engine reproduces known verdicts on Lotka-Volterra, Van der Pol, Lorenz, and a pharmacokinetic model, where it recommends the intravenous arm pharmacologists use, and it ranks the depth scorer of our own benchmark last among four observation designs. On the DiscoverPhysics benchmark, it finds two public worlds whose explanation rubric rewards a distinction no legal experiment can make, and every model there with accurate trajectories failed the explanation grade (15 of 15, against 5 of 9 in identifiable worlds, p = 0.012). On the Alien Universe, a two-body testbed we propose in which a force law switches between a provably non-identifiable and an identifiable protocol, LLM-IDEA on the identifiable protocol reaches discovery depth at least three on 8/8 seeds versus 1/8 without it. An autonomous discovery agent can thus compute, rather than guess, whether a plateau calls for more search, a better experiment of the same kind, or a different kind of experiment.
1 Introduction
An experimental plateau may reflect either limited agent capability or data that cannot identify the target, so further search alone may be futile. LLM-IDEA adds an identifiability engine to distinguish these cases and evaluate experiments and benchmarks.
Certified exhaustion and a three-way plateau triage
- The diagnostic returns a three-way plateau triage—capability limit, resolvable within the design class, or unresolvable in that class—and identifies confounded parameter directions.Its certificate is sound at regular points; the paper also describes a singular-point counterexample to a first-order certificate.Source: §1 · p. 2
The Alien Universe
- The Alien Universe holds the force law fixed while switching between non-identifiable and identifiable observation protocols.The identifiable arm clears a cross-configuration gate, but its prompt discloses a per-body property and runs share distilled memory.Source: §1 · p. 2
The diagnostic turned on the benchmark’s own metric
- The engine ranks the benchmark’s own depth-scorer observation design last among candidate designs for recovering the target exponent.The diagnostic applies the same rank-and-conditioning check to the metric that grades agents.Source: §1 · p. 2
Validation beyond our testbed
- 60 of 62 ODEBench systems with free constants were identifiable at round 0; the RC circuit was certified exhausted, and one harvesting model was resolvable within the protocol.The engine also reproduced settled verdicts on Lotka-Volterra, Van der Pol, Lorenz, and a pharmacokinetic model, where it recommended the intravenous arm.Source: §1 · p. 3
- LLM-IDEA applies its diagnostic to DiscoverPhysics rubrics, finding two worlds where the top explanation grade rewards a distinction no legal experiment can make.Models with accurate trajectories still fail the explanation grade in those worlds.Source: §1 · p. 3
1 Introduction
- The identifiability mathematics is prior work; the contribution is its use as an agent-loop stopping and design rule, while the live-loop demonstration is not a powered measurement.The paper notes confounds in the Alien Universe results.Source: §1 · p. 3
2 Related Work
Related work includes agentic experiment-selection systems, identifiability theory and tools, end-to-end AI scientists, and discovery benchmarks. LLM-IDEA distinguishes its contribution as a class-level exhaustion verdict within an agent loop and a testbed that switches identifiability while holding the law fixed.
2.1 Closed-loop LLM experiment design
- Model Discovery Agent [21] selects designs by value of information, while LLM-ACES [22] uses disagreement; neither returns the class-level verdict that no experiment of a kind can help.CARTOGRAPH [24] uses disagreement Jacobians and Fisher information, but lacks a language-model proposer in its main loop and a design-class exhaustion certificate.Source: §2.1 · p. 3
2.2 Identifiability and Observability
- Prior work defines structural and practical identifiability and develops symbolic, sensitivity-based, and symmetry-based methods; LLM-IDEA operationalizes the class-level question at an agent’s operating point.The symbolic route checks parameter symmetries, not symmetries that also transform states.Source: §2.2 · pp. 3–4
2.3 End-to-End AI scientists
- First-generation AI scientists use generate-and-fit patterns, and the authors say these systems do not test whether a plateau reflects a weak proposer or an answer absent from the data.The AI Scientific Community programme extends these systems to swarms of agentic virtual laboratories.Source: §2.3 · p. 4
2.4 Discovery Benchmarks
- Existing discovery benchmarks provide answer keys and success scores, whereas the proposed testbed holds one law fixed and switches between provably non-identifiable and identifiable protocols.Gravity-Bench varies observation density, while ActiveSciBench grades identifiability strength without exposing it as a switch.Source: §2.4 · p. 4
3 Methodology
The methodology pairs a language-model Council that proposes hypotheses and experiments with deterministic identifiability checks that assess recoverability and guide design selection. A certificate distinguishes plateaus resolvable within a design class from those requiring a change of class.
3.1 The Agentic Council
- The Council drafts hypotheses and candidate designs, then proposes alternatives on non-identifiable plateaus for ranker selection and re-diagnosis.Its five roles share a knowledge graph; the deliberation court made no model calls in the reported runs.Source: §3.1 · pp. 4–5
3.2 The Identifiability Engine
- The engine uses exact symbolic structural tests for rational or ODE models and practical sensitivity-matrix and Fisher-information analysis as a general fallback.The practical route assumes predictions can be evaluated as a function of parameters, including for black-box or non-ODE hypotheses.Source: §3.3 · p. 5
3.3 The Diagnostic
- The diagnostic uses a local sensitivity matrix: full rank indicates local identifiability, a null space exposes confounded parameter combinations, and poor conditioning indicates practical non-identifiability.The analysis is local; probing multiple operating points is the stated remedy when global behavior matters.Source: §3.3 · pp. 5–6
3.4 The Design Ranker
- The heuristic ranker prioritizes recovered parameter directions, uses conditioning as a tie-break, and selects multi-product configurations to break Alien Universe charge-amplitude confounding.On identical candidates, it restored rank as reliably as the compared optimal-design criteria, while A-optimal design chose more precise experiments.Source: §3.4 · p. 6
3.5 Certified exhaustion and plateau triage
- The certificate distinguishes capability limits, in-class resolvability, and certified exhaustion; examples include Alien Universe and oral pharmacokinetic plateaus and a non-certified sequence plateau.It checks whether a confounded direction is invisible across the whole design class, rather than only the current design.Source: §3.5 · pp. 6–7
3.6 When the certificate is a proof
- Class-wide certification is sound at regular points: under constant class rank, a parameter submanifold has identical predictions for every in-class design.Prediction-only scores on in-class designs therefore cannot distinguish parameters along a certified direction.Source: §3.6 · p. 7
3.7 Algorithm and usage
- Certification checks both first-order invisibility and stable class rank; closed-form models can receive proofs, while black-box numeric checks provide evidence over sampled designs.The regularity check guards against falsely certifying singular points where nearby parameters have higher rank.Source: §3.7 · pp. 8–9
4 The Alien Universe Testbed
The Alien Universe is a synthetic two-body testbed designed to switch whether a hidden per-body property is identifiable from observations. Its sections define the dynamics and scoring ladder, describe the switch and agent inputs, and qualify how results should be interpreted.
4 The Alien Universe Testbed
- The testbed holds its force law fixed while switching between a single configuration, where a hidden property is non-identifiable, and a family that makes it identifiable.The diagnostic verifies a rank drop and named null direction in the degenerate arm, and full rank in the observable arm; the family prompt also differs, so this is not a clean switch.Source: §4 · p. 9, §4.3 · pp. 11–12
4.1 The Dynamical System
- The invented law combines a hidden charge-dependent force with tangential inertia, which affects motion perpendicular to the bodies’ joining axis.The radial terms have different distance exponents, and the charge enters through the product of the bodies’ charges.Source: §4.1 · pp. 10–11
4.2 The depth ladder
- Capability is scored as consecutive depth through L1–L4, from the dominant radial law to the hidden force law; the target is structural discovery, not recovery of individual charges.The charges occur only through their product, while L5 and L6 are reported but are not part of the capability measure.Source: §4.2 · p. 11
4.3 The observability switch
- With one configuration, only the lumped quantity βs1s2 is constrained; a family reveals per-system products, but cannot identify the two charges separately or separate β from unknown charges.The single-configuration protocol has a discoverable ceiling of depth 2; rescaling all charges and β together leaves every system unchanged.Source: §4.3 · p. 12
4.4 What the agents see
- Agents receive times, positions, velocities, and masses, but not charges, couplings, the conserved quantity, parity symmetry, or the closed-form law; scoring uses held-out initial conditions.The held-out evaluation checks symbolic equivalence, trajectory agreement, and qualitative shape.Source: §4.4 · p. 12
4.5 Construct-validity note
- Because the Alien Universe is synthetic and authored, its measured depth indicates capability under controlled observability, not new knowledge about nature.The invented law makes the switch exact and the answer key machine-checkable.Source: §4.5 · p. 12
- The single-configuration L1 and L2 gates test an exponent near 3.5, not whether each proposal’s own correction term fits; proposal-coupled L2 rescoring removes the reported architecture separation.Over the orbit range, the solo agent’s r^-3 term reduces the residual marginally more than the Council’s r^-3.4 term.Source: §4.5 · pp. 12–13
5 Results
The results test LLM-IDEA on external systems and the Alien Universe, then demonstrate a live loop that can select a resolving design or certify exhaustion. The Alien treatment improves discovery depth, while transfer results show that band-gated outcomes can shift as the fitting engine is corrected.
5.1 Protocol
- Measured cells used eight paired seeds, preregistered predictions, paired bootstrap intervals, Wilcoxon tests, Cliff’s delta, and Holm-Bonferroni correction.The protocol also reports exploratory analyses separately and records failed predictions.Source: §5.1 · p. 13
5.2 The diagnostic on systems we did not build
- Full-state Lotka-Volterra, Van der Pol, and Lorenz were structurally identifiable; an intravenous arm restored pharmacokinetic-model rank to 4 of 4.For oral sampling, the pharmacokinetic diagnostic found a confounded V,F direction and recommended the intravenous design pharmacologists use.Source: §5.2 · pp. 14–15
- The diagnostic’s Lotka-Volterra prey-only verdict assumes a known initial condition; a higher-order confound exposed a blind spot in the original symbolic checker.The checker was extended to detect constant-coefficient linear dependence, but the diagnostic remains limited by its assumptions about initial conditions.Source: §5.2 · pp. 14–15
5.3 Observability lift and second-regime transfer
- 8/8 treatment seeds reached depth ≥3 versus 1/8 controls; treatment combined a multi-charge family, residual feedback, and an exponent-fitting battery.The factorial analysis attributes the cross-configuration layer to the family and the correction-exponent layer to the fitting engine.Source: Figure 2 · p. 15, §5.3 · pp. 15–16
- L3 held at 7/8 with either fitting-engine state and met 6/8 after both visible channels were corrected; band-gated layers fell 5/8, 3/8, and 2/8.The corrected battery reported a data-optimal exponent near 2.1, while the L2 band was centered on the truth at [2.2, 2.8].Source: Figure 3 · p. 16, §5.3 · p. 17
- 16/16 runs with the family passed L3 versus 4/16 without it; 16/16 with the fitting engine passed L2 versus 8/16 without it.The family made the cross-configuration gate passable, while the fitting engine delivered the correction exponent.Source: §5.3 · p. 16
5.4 The loop closes, and can stop with a certificate
- 3 of 5 scored director calls requested a multi-charge family and returned SOLVED; the other two requested an initial condition and exhausted after three rounds.With the certificate, the initial-condition branch terminates at round 0 by proof because the scaling direction is a symmetry of every single-configuration design.Source: §5.4 · pp. 17–18, Figure 4 · p. 17
- Michaelis-Menten substrate time courses gave rank 2 of 3, and an enzyme-concentration assay restored full rank; a narrow-window sequence ansatz resolved in one round.The certificate for the Michaelis-Menten case was platform-dependent; the sequence ansatz collapsed through the observed window, not across the design class.Source: §5.4 · p. 18
5.5 The diagnostic turned on the benchmark’s own metric
The diagnostic exposes when benchmarked targets exceed an observation design’s resolving power, while its design audits distinguish feasibility from precision and reveal benchmark-specific limits.
5.5 The diagnostic turned on the benchmark’s own metric
- The depth scorer’s held-out design ranked last among four, while the combined design D4 ranked first but still could not pin exponent 3.5 against steeper exponents.D4’s indifference band extended to 5.50, although it excluded 3.Source: §5.5 · p. 19
5.6 Architecture versus tools: a gate-dependent separation
- Under the registered band gate, Council reached depth 3 and Solo depth 1 on every seed, but the separation vanished under the proposal-coupled gate.The call-matched confirmatory test missed its threshold; descriptively, terminal accuracy tracked delivery of fitting-engine feedback.Source: §5.6 · p. 20
5.7 Transfer to integer sequences with a machine-checked certificate
- On integer sequences, geometric-spread sampling restored rank 3 of 3 and exact recovery; the certifier is sound when it certifies but incomplete when exact solving fails.An out-of-basis refusal does not establish that no closed form exists.Source: §5.7 · p. 20
5.8 The diagnostic on the benchmark a competitor uses
- 60 of 62 ODEBench systems with free constants were identifiable at round 0; the RC circuit was certified exhausted, while harvesting was resolvable with one added initial condition.For the RC circuit, no initial condition or time window in the tested class can separate the constants.Source: §5.8 · p. 21
5.9 Which experiment to run: a head-to-head of selection criteria
- A-optimal design had the lowest median parameter error (1.60 × 10−3) and alone significantly beat the ranker; disagreement and random selection were weakest.The authors note that this outcome measure is the quantity A-optimal design minimizes, while a volume measure would favor D-optimal design.Source: §5.9 · pp. 22–23
5.10 An identifiability audit of a published discovery benchmark
- 15 of 15 accurate-trajectory cells in two certified-equivalence worlds failed explanation grading, versus 5 of 9 identifiable-world cells (p=0.012).The two rubrics reward naming a fractional operator rather than a power law, although the model families are equivalent for every legal experiment.Source: §5.10 · pp. 23–24, Table 2 · p. 24
6 Limitations
The paper qualifies its identifiability claims with limits on certification, experimental design, and evaluation scope. These boundaries affect what its verdicts and benchmark comparisons establish.
6 Limitations
- The Alien Universe lift is a joint effect of protocol and task text, and the single-configuration arm cannot pass the cross-configuration gate by construction.A clean replication would hold task text fixed across arms.Source: §6 · p. 24
- Shared judging criteria can carry across runs and arms, violating the live-agent cells’ independent-seed assumption and making their p-values optimistically biased by an unmeasured amount.The proposed replication control is to clear or run-scope the criteria table between seeds.Source: §6 · pp. 24–25
- The Alien lift uses eight seeds on authored laws, and its factorial spans provider rosters, although each seed’s two arms shared a roster.Depth on the authored laws does not establish knowledge about nature; comparison with original treatment and control cells is not roster-matched on half the seeds.Source: §6 · p. 25
- The diagnostic establishes local, not global, identifiability and can miss a second discrete solution; its verdict also depends on which quantities are declared known.The pharmacokinetic flip-flop case illustrates the global limitation, while Lotka-Volterra changes verdict when the initial condition is unknown.Source: §6 · p. 25
- The ranker is not the best precision tie-break: its ODEBench-only comparison rarely tested rank restoration, while numerical rank changed under parameter scaling.A-optimal design was favored for precision, and the harvesting model was rank-deficient in raw coordinates but full-rank and ill-conditioned in log-parameters.Source: §6 · p. 25
- The live-loop result is descriptive, based on five scored calls and two moves; model families were not varied, and the identifiability mathematics is prior work.The live director did not author designs, and the plateau was constructed rather than harvested.Source: §6 · p. 25
- The integer-sequence experiment re-derives textbook facts, searched only OEIS term windows, and makes no mathematical contribution.The certifier is sound at literal zero but incomplete otherwise; one true identity was returned unverified.Source: §6 · p. 25
- Symbolic certification requires a closed-form observation model with symbolic design controls; sampled numeric null directions are evidence at those designs, not proof over the class.A narrowly declared design class can report exhaustion even when a wider class would resolve the parameter; singular operating points and platform-dependent thresholds also constrain the numeric route.Source: §6 · p. 26
7 Conclusion
LLM-IDEA couples a language-model proposer to a deterministic identifiability engine that diagnoses confounded directions and certifies when the current design class cannot resolve them. Across external systems and benchmarks it reproduces known verdicts and identifies unresolvable rubric distinctions, while the identifiable Alien Universe protocol reaches depth at least three on 8/8 seeds.
7 Conclusion
- 60 of 62 ODEBench systems with free constants were identifiable at round 0; the RC circuit was certified exhausted, and one harvesting model was resolvable by one added initial condition.The engine also reproduced settled verdicts on Lotka-Volterra, Van der Pol, Lorenz, and an oral pharmacokinetic model, recommending the intravenous arm.Source: §7 · p. 26
- On the Alien Universe, the identifiable protocol reached depth at least three on 8/8 seeds, versus 1/8 without it; the lift was a joint effect of protocol and task text.The factorial attributed the cross-configuration layer to the family of systems and the correction exponent to the fitting engine.Source: §7 · p. 26
- DiscoverPhysics contains two public worlds whose rubrics reward distinctions no legal experiment can make; accurate-trajectory models failed the explanation grade in 15/15 cells there, versus 5/9 in identifiable worlds.The engine also ranked the depth scorer’s observation design last among four.Source: §7 · p. 26
- LLM-IDEA’s distinction from MDA, LLM-ACES, and CARTOGRAPH is a certified verdict that no experiment of the current kind can help, not superior design selection.Its rank-then-conditioning heuristic was matched by D-optimal design and expected information gain, while A-optimal design performed better on precision.Source: §7 · p. 26
- The verdict turns a plateau into a choice among continued search, a better same-kind experiment, or changing the experiment class.The three-way triage is sound at regular points of the class; the implementation detects and refuses at singular points.Source: §7 · p. 26
8 Reproducibility and Code
The paper describes integrity checks, run records, and release materials intended to support reproducibility. The released package includes the diagnostics, simulator, scorers, and run records.
8 Reproducibility and Code
- The validation pipeline uses no language model, and an automated source check verifies that scoring code imports no model client.Analyses of live-agent measurements, ODEBench, criterion comparisons, and DiscoverPhysics were pre-registered; runs were serialized with seeds, model versions, prompt hashes, and provider request identifiers.Source: §8 · pp. 26–27
- Run records index the experiment logs, and supplementary files provide per-task and per-claim tables for the criterion comparison and DiscoverPhysics audit.The ODEBench sweep used a pinned LLM-ACES checkout and released its per-system verdicts, null directions, certificates, and best designs.Source: §8 · p. 27
- The diagnostics package, Alien Universe simulator, scorers, and run records are released at the project repository’s v1.0 tag.The release is identified as https://github.com/Suraskey/llm-idea.Source: §8 · p. 27
A Architecture versus tools in detail
The appendix provides the design, tables, and run accounting behind Section 5.6.
A Architecture versus tools in detail
- The appendix contains the design behind Section 5.6.Source: A Architecture versus tools in detail · p. 33
- The appendix contains the tables behind Section 5.6.Source: A Architecture versus tools in detail · p. 33
- The appendix contains the run accounting behind Section 5.6.Source: A Architecture versus tools in detail · p. 33
A.1 The equal-wall-clock comparison
The section compares a five-agent Council with a single generalist on the same flagship model and tool stack, varying only the agent configuration. The Council’s paired advantage was consistent across all eight seeds, although the comparison equalized wall-clock time rather than model calls.
A.1 The equal-wall-clock comparison
- All eight paired differences were 2; one-sided Wilcoxon p=1/28=0.0039, Holm-adjusted p=0.0117, and Cliff’s delta 1.0.The equal-wall-clock comparison used eight seeds per arm on the same flagship model.Source: Table 3 · p. 33, §A.1 · p. 33
- The per-layer vectors differ on exactly one layer, the correction-exponent layer, which the paper treats as the source of the separation.Source: §A.1 · p. 33
- In twenty minutes, the Council made 92–169 model calls per run, versus 10–14 for the single agent.The design equalized wall-clock time only.Source: §A.1 · p. 33
A.2 The call-matched control
The call-matched control fixed the tool stack and imposed model-call caps, with three registered budget cells. At 15 calls the arms tied at a starvation floor; at 40 calls a directional gap appeared, while the 130-call cell yielded only limited valid comparisons.
A.2 The call-matched control
- The control fixed the tool stack and imposed a hard model-call cap; only runs terminating on the cap counted toward a budget cell.The registered call budgets were N=15, N=40, and N=130.Source: §A.2 · p. 33
- At N=15, both arms tied at depth 1 on all eight seeds (Cliff’s delta 0), a registered starvation floor with no architecture inference.Fifteen calls gave roughly three per Council agent, too few to finish deliberation.Source: §A.2 · p. 33
- At N=40, the Council reached depth 3 or better on five of eight seeds (mean 2.375), while the single agent stayed at depth 1 on all eight (mean 1.0).The contrast was compute-matched: all runs terminated on the cap with 39–43 realized calls. The registered Holm-adjusted statistical clause was unsatisfiable after the N=15 all-tie result, so the authors claim only a directional effect.Source: Table 4 · p. 33, §A.2 · p. 34
- At N=130, four cap-valid Council runs reached depth 3, while the single-agent runs were sub-budget and remained at depth 1 descriptively.The single-agent runs stopped at 109–125 calls below the cap; the authors make no broader claim for this cell.Source: §A.2 · p. 34
A.3 The proposal-coupled rescore
A proposal-coupled rescore removes the architecture separation seen under the band gate: both arms pass the correction layer and reach the same depth. The authors attribute the original separation to the gate’s scoring rule.
A.3 The proposal-coupled rescore
- Under a gate fitting each proposal’s own correction exponent, the separation dissolves to Cliff’s delta 0, and both arms pass the correction layer at the same depth.The band gate instead credits correction terms within a fixed window of the true exponent without per-proposal numerical fitting; the rescore uses stored records and no model call.Source: §A.3 · p. 34
B Closed-form derivations and search trail
The section describes a loop that derives and certifies three identities, relates two sums to discrete-arcsine moments, and documents the boundaries of its search and certification. The authors also note that OEIS-only searches cannot establish that the forms were absent from the literature.
B Closed-form derivations and search trail
- Two sums are expressed as moments of the discrete arcsine law for the positive-side time of a simple symmetric random walk of length 2n.The second raw moment is given as n(3n+1)/2, with the second moment also appearing in eq. (4.3) of [77].Source: B Closed-form derivations and search… · pp. 34–35
- The three identities follow from (1−4x)^−1/2 and the operator x d/dx; the passage relates special cases to OEIS sequences and prior work [78], [79],.Source: B Closed-form derivations and search… · p. 35
- The loop derived and certified three identities by solving for coefficients from five, six, and seven exact samples, then certifying the results.The certifier maps family definitions and candidate forms into generating-function space and accepts a result only when Sympy simplifies their difference to literal zero.Source: B Closed-form derivations and search… · pp. 34–35
- Target selection used OEIS term searches at three window types, and the authors explicitly do not treat OEIS absence as evidence of literature absence.They report that the moment identities occur in probability literature and no longer describe the forms as unpublished.Source: B Closed-form derivations and search… · p. 35
- A piecewise object matching on 60 of 60 integers was still reported UNVERIFIED, while an out-of-basis control was refused without proving that no closed form exists.Source: B Closed-form derivations and search… · p. 35
C Prompts given to the language models
This appendix makes the language models’ inputs auditable by reproducing prompts, linking calls to system-prompt hashes, and showing rendered requests. It describes prompts for inferring two-body motion laws from trajectory data, including variants that compare systems or refine a proposed law using feedback, with constrained symbolic outputs and later residual and controlled-experiment feedback.
C Prompts given to the language models
- Every experimental prompt is reproduced verbatim with a SHA-256 that can match it to logged model calls; the scorer and Argus overseer make no model calls.Role prompts use named template slots, and deterministic fitting feedback enters the Agora-context slot.Source: C Prompts given to the language models · p. 35
C.1 Which prompts were called
- The call log covers 2026-05-30 onward, groups calls by system prompt, excludes two early negatives, and omits embedding calls; Table 5 gives the window’s scope.The five Council roles are registered as Plato, Aristotle, Diogenes, Heraclitus, and Parmenides; the body uses theorist, empiricist, contrarian, synthesizer, and replicator.Source: §C.1 · p. 36, Table 5 · p. 36
- The registry has 19 prompts without hashed calls in the window; the director prompt nevertheless made six EXP-078 calls stored without a system-prompt hash.The other unhashed prompts were unused by results and are included in the code release.Source: §C.1 · p. 36
C.2 The system prompts
- The five Council prompts encode distinct roles: data-fitting empiricism, simplicity, forced contradiction, cross-agent synthesis, and evidence-backed refinement.Aristotle prioritizes held-out residuals; Plato favors the simplest supported form; Diogenes requires a contradiction; Heraclitus synthesizes other agents; Parmenides favors refinement and existing support.Source: §C.2 · pp. 36–39
- Heraclitus’s contrastive-distillation prompt asks for generic, observable criteria distinguishing a better hypothesis from a worse one, avoiding subjective judgments.It allows at most five criteria and requires “NONE” rather than an empty response if no clean distinction is found.Source: §C.2 · p. 39
- The solo prompt combines first-principles derivation, fitting, self-criticism, and parsimony, while directing the model to investigate structured residuals beyond an obvious law.Source: §C.2 · p. 40
C.3 Two fully rendered requests
- Two rendered EXP-089 requests illustrate the family-only prompt feed and the engine-only request’s two deterministic feedback blocks.The examples are Aristotle’s first call in a family-only run and Parmenides’s last call in an engine-only run; sampled trajectory lines are shortened in the appendix.Source: C Prompts given to the language models · p. 35, §C.3 · p. 40
- One request presents four systems governed by the same law but differing in an unobserved per-body property, directing the model to compare them to infer how that property affects the law.The task asks for one symbolic acceleration expression that may include the latent property.Source: §C.3 · pp. 41–42
- The requests require literal CouncilHypothesis JSON with a posterior over up to five candidates, matching candidate keys, and an optional reasoning trace capped at 500 characters.The initial prompt encourages broader candidate consideration, whereas the refinement prompt directs probability mass toward evidence-supported candidates.Source: §C.3 · pp. 42, 44
- The Parmenides role prompt favors refining and replicating supported hypotheses, while its judging criteria emphasize mechanisms, relative velocity, controlled experiments, testable constants, and precise exponents.It also instructs the model to concentrate posterior mass on candidates with at least two supporting Agora nodes.Source: §C.3 · pp. 42–43
- A second request asks the model to infer a governing two-body motion law from one system’s observed trajectory, presented at 12 sampled timepoints.The task describes the law as unknown and requests one symbolic expression for acceleration.Source: §C.3 · p. 43
- Both request variants constrain symbolic forms to specified observable names and a plain-ASCII right-hand-side expression without an equation prefix or equals sign.Permitted functions include sin, cos, exp, log, and sqrt.Source: §C.3 · p. 43
- The iterative prompt says the current best law explains about 94% of observed motion and asks for a steeper continuous inverse-power term plus an oscillatory component to capture systematic leftovers.It warns against re-proposing a frequently suggested law shape and tells the model to build on the search history.Source: §C.3 · pp. 43–44
- In controlled-experiment feedback, a correction fit on cleaner data has an inverse-power exponent of about 3.4 and removes about 100% of the inverse-square-only leftover.The prompt asks the model to consider committing to an inverse-square term plus a 1/r**3.4 correction.Source: §C.3 · p. 44