PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S. Weld, Peter Clark

arXiv:2503.22708v1cs.AIcs.CL

TL;DR

Autonomous scientific discovery systems face constrained design spaces and limited code evaluation. CodeScientist uses genetic search over literature and codeblocks to run automated experiments, identifying 19 potential discoveries, 6 of which met minimum soundness and incremental-novelty thresholds.

  • Problem

    Evaluating autonomous scientific discoveries is difficult because gold-standard outcomes are generally unavailable, requiring alternative assessments.

    Source: §1 · p. 1

  • Method

    CodeScientist performs genetic search over combinations of research literature and codeblocks, then builds, runs, and analyzes the resulting software experiments.

    Source: §1 · p. 1, §6 · p. 9

  • Results

    19 potential discoveries were identified, of which 6 met minimum thresholds for scientific soundness and incremental novelty after external review and internal code review.

    Source: §1 · p. 1, §6 · p. 9

  • Takeaways & Limitations

    The candidate discoveries broadened the explored scope to include novel tasks, agents, metrics, and data.

    Source: §1 · p. 1, §6 · p. 9

    Low-sample experiments can produce false positives and miss effects smaller than the largest detectable effects.

    Source: §7 · p. 10

Abstract

from arXiv · show

Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.

1 Introduction

CodeScientist introduces an ASD system that uses genetic search over combinations of literature and code to explore agents and virtual environments at scale. It reports 19 suggested discoveries, 6 meeting minimum soundness and incremental-novelty thresholds after expert review, spanning tasks, agents, metrics, data, and challenged assumptions.

  • CodeScientist uses genetic search over combinations of research literature and code to conduct hundreds of experiments in agents and virtual environments.The system is designed to broaden ideation and experiment execution beyond more restricted ASD approaches.Source: §1 · p. 1
  • Of 19 suggested discoveries, 6 met minimum thresholds for scientific soundness and incremental novelty after domain-expert review.Source: §1 · p. 1
  • The discoveries span new tasks, agents, metrics, data, and challenges to assumptions, broadening ASD beyond standardized ML benchmark optimization.Existing systems simplify discovery by restricting searches to prewritten-code variants, domain-specific languages, or restricted problems.Source: §1 · p. 1
  • Evaluating ASD discoveries is methodologically difficult because gold-annotated outcomes are generally unavailable at the edge of human knowledge, motivating rigorous manual review or alternatives.Source: §1 · p. 1
  • The system incorporates human-provided papers and short example codeblocks, including an LLM-call codeblock, as inputs to its experimentation process.Source: §1 · p. 2

Ideation

CodeScientist generates candidate ideas by mutating input papers and codeblocks with an LLM, then uses human selection and comments to refine promising proposals. One example examines whether self-assessed LLM confidence correlates with accuracy in state-prediction tasks.

Generate batch of candidate ideas

  • An LLM-as-a-mutator takes papers and codeblocks as input and outputs candidate research ideas.Source: Generate batch of candidate ideas · p. 2
  • Idea #314 proposes testing whether self-assessed LLM confidence correlates with accuracy in state-prediction tasks using 200 TextWorldExpress state-prediction pairs.The pilot focuses on boolean property predictions with conference scores.Source: Generate batch of candidate ideas · p. 2
  • Humans select a subset of promising ideas and provide brief comments on each.For Idea #314, feedback recommends an LLM-as-a-judge metric instead of raw string matching to handle presentation variations more robustly.Source: Generate batch of candidate ideas · p. 2

Planning

CodeScientist’s planning workflow turns generated ideas into executable experiments by combining domain environments, LLM calls, evaluation procedures, and reusable codeblocks. For Batch Idea #314, repeated runs supported the hypothesis, followed by reporting, cross-experiment analysis, and human verification of code and results.

Create experiment plans

  • Batch Idea #314 uses TextWorldExpress CookingWorld environments with 3 ingredients, 2 distractors, no doors, and gpt-4o-mini for all LLM calls.The plan collects state predictions with confidence scores and uses an LLM-as-a-judge to score prediction accuracy from 0 to 1.Source: Create experiment plans · p. 2
  • The experiment runs a pilot of 3 episodes with 10 steps each and a full evaluation of 50 episodes with 25 steps each.The plan includes codeblocks for LLM calls, TextWorldExpress, plotting, robust inferential statistics, and logging/debugging.Source: Create experiment plans · p. 2
  • The generated artifact implements model, evaluation, and experiment-running functions that loop over episodes, instantiate TextWorldExpress, and evaluate one episode at a time.The resulting analysis compares accuracy with confidence.Source: Create experiment plans · p. 2
  • Overall results support the hypothesis across the reported independent experiment runs.Runs 1, 2, 3, and 5 are each labeled as supporting the hypothesis.Source: Create experiment plans · p. 2
  • Human examination verifies the generated code and results as part of the workflow’s evaluation of interesting experiments.The broader system also uses external review, replication, and code review to vet candidate discoveries.Source: Create experiment plans · pp. 2–3
  • The workflow spans ideation, planning, code-based experiment execution, result reporting, and cross-experiment meta-analysis.Figure 1 presents these stages as the core CODESCIENTIST discovery workflow.Source: Create experiment plans · p. 2

2 Related Work

Prior ASD systems vary in ideation, experiment construction, execution, and domain, while discovery evaluation remains limited by costly manual review and imperfect automated proxies. CodeScientist combines literature-based ideation with executable codeblocks and emphasizes faithfulness between reported discoveries and implemented code.

Ideating and Executing Research

  • CodeScientist combines literature-based ideation with partial conditioning on executable codeblocks, balancing open-endedness with ideas its experiment builder can implement.Prior approaches instead mutate existing experiment code or restrict ideation to task-specific templates (Hemberg et al., 2024; Liu et al., 2024; Ifargan et al., 2024).Source: §2 · p. 3

Evaluating Discoveries

  • Automated discovery evaluation is limited: manual code review is costly, while LLM-as-a-judge paper scores may disagree with human judgments and be fooled by superficial qualities (Lu et al., 2024a; Schmidgall et al., 2025; Li et al., 2024).Automatic evaluation is possible in limited domains with molecular simulators, instrumentation, or measurable objectives (Qi et al., 2024; Jansen et al., 2024; Huang et al., 2024).Source: §2 · p. 3
  • Faithfulness is critical because a system’s claimed discoveries can differ substantially from what its code implements, making paper-only conference-style Likert assessments insufficient.The paper identifies this mismatch as a central limitation of evaluating science-as-code discoveries.Source: §2 · p. 3

3 System Overview

CodeScientist implements a prompt-driven workflow that transforms curated papers and reusable codeblocks into ideas, executable experiments, reports, and consistency analyses. Human selection and comments constrain ideation, while iterative sandboxed execution and repeated runs support tractable, reproducible experimentation.

3 System Overview

  • The system comprises five major steps—ideation, planning, experiment building, reporting, and meta-analysis—implemented primarily as language-model prompts.Example prompts and additional implementation details are provided in Appendix F.Source: §3 · pp. 3–4

3.1 Paper Corpus and Codeblocks

  • CodeScientist requires a human-curated paper list and vetted codeblocks for common tasks such as calling an LLM, implementing a ReAct agent, and loading benchmarks.These inputs define the literature and executable capabilities available to the system.Source: §3.1 · p. 4

3.2 Ideation

  • The ideator combines two papers with codeblock summaries to generate structured candidate ideas, then applies crossover and mutation before human selection and comments.Each idea specifies hypotheses, variables, metrics, baselines, pilot design, and anticipated implementation resources.Source: §3.2 · p. 4

3.3 Planning

  • The planner converts each selected idea and expert comments into an operational artifact-and-experiment plan and identifies the required codeblocks.Plans specify implementation and experimental details such as base models and hyperparameters.Source: §3.3 · p. 4

3.4 Experiment Construction and Execution

  • The experiment builder generates code, executes it in an instrumented cost-limited sandbox, and iteratively reflects and debugs until completion or a hard limit.It first attempts an inexpensive pilot and automatically scales successful pilots to the full experiment; removing expert comments reduced sound-and-novel discoveries by approximately one third.Source: §3.4 · pp. 4–5

3.5 Reporting

  • Reporting converts completed plans, code, results, and logs into a detailed LaTeX report and a short summary that categorizes hypotheses as confirmed, rejected, or inconclusive.The short summary helps users prioritize which high-throughput results warrant reading in full.Source: §3.5 · p. 5, Table 2 · p. 5

3.6 Meta-Analysis

  • Meta-analysis reruns each idea and plan N times to examine consistency across variable implementations and produces a meta-analysis report.Repeated runs address language-model variability and failures in complex scientific code generation.Source: §3.6 · p. 5

4 Discovery Experiments

CodeScientist ran a large-scale discovery pipeline over agent architectures and virtual environments, generating and testing candidate ideas before subjecting promising discoveries to external and internal evaluation. The system flagged 19 of 50 ideas for human inspection, using reviews of soundness, novelty, code, logs, and replication attempts.

4 Discovery Experiments

  • The experiment corpus combined 57 recent papers with 10 code snippets covering language-model calls, ReAct agents, plotting, statistics, knowledge graphs, and benchmark environments.The snippets served as vetted implementations and demonstrations of sandbox-specific coding requirements.Source: §4 · pp. 5–6
  • Each of 50 selected ideas received five experiment-builder attempts, producing 250 total experiment runs.The ideas were selected from approximately 2000 candidates generated from 200 randomly selected paper combinations, with expert selection of 50 viable and sufficiently distinct ideas.Source: §4 · p. 6

4.1 Candidate Discoveries

  • 19 of 50 ideas were flagged for human inspection after at least one of five experiment runs produced “interesting” results.These candidate discoveries were then evaluated through separate external and internal reviews.Source: §4.1 · p. 6
  • External review converted ratings from three domain experts into binary soundness and novelty scores, counting a discovery as passing when most reviewers met both minimum thresholds.Ratings meeting minimum soundness included clearly sound, likely sound, or minor-concern assessments; incremental novelty or above met the novelty threshold.Source: §4.1 · p. 6
  • Internal review examined code and experiment logs and attempted replication with more samples, allowing a domain expert to reject discoveries that failed detailed examination.This review functioned as a veto over discoveries that appeared genuine from the paper and external review.Source: §4.1 · p. 6

Human Reviewers

Human evaluation reduced 19 CodeScientist-flagged discoveries to 6 that met minimum soundness and incremental novelty criteria in both external review and internal code, log, and replication checks. The surviving discoveries mostly introduced tasks, benchmarks, metrics, methods, or challenged assumptions rather than merely improving benchmark performance.

Human Reviewers

  • The accepted discoveries included multi-stage environment generation improving fidelity, poor LLM combinatorial optimization performance, low action-success prediction ability, and a graph-memory ReAct agent outperforming a ReAct baseline in DISCOVERYWORLD.The action-prediction finding was qualified by using only the current observation and imperfect LLM-as-a-judge labels; the graph memory was relatively simple object-property memory.Source: Human Reviewers · p. 7
  • 6 of 19 discoveries (32%) passed both external review and internal review after code examination, experiment-log inspection, and replication attempts.External reviewers judged 13 of 19 (68%) as meeting minimum soundness and novelty criteria, but the internal reviewer rejected 7 of those 13.Source: Table 4 · p. 7, Human Reviewers · p. 8
  • The six accepted discoveries spanned new tasks, benchmarks, metrics, methods, and challenges to assumptions, with most not simply improving benchmark performance.Examples included low correlation between LLM self-assessed confidence and state-prediction accuracy, representation-complexity effects on state prediction, poor action-success prediction, and weak performance on a specific combinatorial optimization problem.Source: Human Reviewers · pp. 7–8
  • Human evaluation rejected 13 of 19 discoveries: 6 involved overly simplistic baselines or models, 3 effects disappeared with larger samples, 2 had major implementation errors, and 1 lacked sufficient detail for soundness evaluation.Source: Human Reviewers · p. 8

5 Discussion

CodeScientist’s discovery quality is limited by duplicate ideas, failed or unfaithful experiments, weak research practices, and substantial human intervention. Fully automated operation is possible but currently has low discovery rates, high variance, and prohibitive cost for statistically powered evaluation.

Idea Diversity

  • Although discoveries span diverse categories, most generated ideas are duplicates or mechanical variations, requiring manual filtering despite embedding-based uniqueness filtering.The discussion identifies increasing idea diversity and utility as a target for future ideation and evaluation systems.Source: §5 · pp. 8–9

Experiment Builder Failures

  • 59% of planner-designed experiments fail implementation, chiefly because debugging reaches its iteration limit (32%) or experiments hit time limits (18%).A further 9% of failures arise when generated experiment code exceeds 8K output tokens.Source: §5 · p. 8

Unfaithful Experiments

  • Human inspection found an apparently successful graph-based agent never used its constructed graph and instead selected actions randomly, making automated detection of unfaithful results necessary.Such errors are difficult and laborious to detect and can produce false positives or false negatives.Source: §5 · p. 8

Adherence to best research practices

  • Generated experiments often violate research best practices by evaluating on training data, miscalculating or misinterpreting statistics, or using weak baselines that can yield spurious significance.Brief domain-expert comments currently help correct some methodological problems before costly experiments, while reducing human effort remains a future goal.Source: §5 · p. 9

Fully-automated Mode

  • Fully automated runs produced 2 internally reviewed candidate discoveries from 100 ideas (2%), while removing expert comments yielded 4 of the 6 benchmark discoveries.The authors caution that this small-sample reduction may reflect missing expert comments or natural system variance, and that high variance and cost make statistically powered ablations impractical.Source: §5 · p. 9

6 Conclusion

CodeScientist performs semi-automated scientific discovery by jointly searching literature and codeblocks, then constructing and analyzing software experiments. It identified 19 potential discoveries, six meeting minimum soundness and incremental-novelty thresholds after external review and internal code review, spanning tasks, agents, metrics, and data.

System and scope

  • CodeScientist jointly searches combinations of research literature and domain codeblocks before building, running, and analyzing software artifacts in experiments.The system's ideator uses genetic search over literature and codeblocks in the domain of agents and environments.Source: §6 · p. 9

Findings

  • 19 potential discoveries were identified, including 6 judged to meet minimum thresholds for scientific soundness and incremental novelty after external conference-style and internal code review.The evaluation included both external review and internal code review.Source: §6 · p. 9
  • The candidate discoveries span novel tasks, agents, metrics, and data, broadening the scope of ideas explored by other systems.This qualitative assessment concerns the set of generated discoveries rather than a single benchmark result.Source: §6 · p. 9

Future research

  • The work analyzes common failure modes and releases CodeScientist as open-source to facilitate future research in automated scientific discovery.These activities are presented as resources for advancing subsequent research.Source: §6 · p. 9

7 Limitations … B External Reviewer Rubric

The paper identifies limitations in cost, validation, discovery impact, and ideator recall, while the external-review rubric evaluates soundness and novelty using categorical criteria. These limitations frame the six validated discoveries as preliminary, generally incremental candidate discoveries.

7 Limitations

  • CodeScientist’s experiments average approximately $4 and 2 hours, trading low cost and rapid iteration for false positives and missed small effects.Scaling computation could address this limitation, while smarter investment in promising experiments may reduce budget requirements.Source: §7 · p. 10
  • More than half of potential discoveries were rejected by internal code review for serious issues, including code that simplified or misrepresented the intended mechanism.Such errors can be difficult to detect because they may involve only a few lines within long programs, motivating automated review mechanisms.Source: §7 · p. 10
  • The six externally and internally reviewed discoveries are framed as “candidate discoveries” because inaccurate mechanisms may remain undiscovered despite code examination and larger-scale reruns.The task’s adversarial nature makes validation more difficult than ordinary article review.Source: §7 · p. 10
  • The six expert-validated discoveries are likely incremental rather than transformational, and the study cannot determine whether higher-impact discoveries require greater scale or a different discovery process.External reviewers’ ratings indicate that each candidate was, at best, incrementally novel.Source: §7 · p. 10
  • The ideator’s discovery recall cannot be estimated because the study manually filtered a larger idea set to 50 seemingly implementable and distinct ideas.Pilot ideas included near duplicates, incorrect metrics, and difficult implementations, so the selected set does not support claims about the proportion becoming human-verified discoveries.Source: §7 · p. 10

B External Reviewer Rubric

  • External reviewers assessed each Table 4 paper on categorical scientific soundness and novelty scales, providing ratings, justifications, and brief descriptions of contributions and claims.Soundness concerns whether methods and evidence support the claims, while novelty concerns originality relative to existing work.Source: B External Reviewer Rubric · p. 14
  • The soundness rubric ranges from clearly sound and likely sound to minor concerns and unsound, depending on methodological rigor, reliability, and flaws undermining claims.Minor concerns may affect measurements without changing overall conclusions, whereas unsoundness undermines credibility.Source: B External Reviewer Rubric · p. 14
  • The novelty rubric distinguishes highly novel work, significant or minor incremental variation, and work that is not novel because it exists in exact form.Examples range from new concepts or frameworks to modest adaptations and direct replication.Source: B External Reviewer Rubric · p. 14

C Meta-Analysis Categorization Criteria … Meta-Analysis Prompt

This section defines the categorization criteria and prompts governing feasible idea generation, experiment planning, debugging, execution, reporting, and meta-analysis. Together, they specify a staged, reproducible workflow requiring faithful implementation, explicit resource and hypothesis reporting, validation, and structured outputs.

C Meta-Analysis Categorization Criteria

  • Results are classified as consistent when at least 4 of 5 independent runs support or reject the hypothesis, mixed otherwise, or limited when at most 2 runs complete.The limited category applies regardless of whether completed experiments support, reject, or inconclusively address the hypothesis.Source: C Meta-Analysis Categorization Criteria · pp. 14–15

D Example Domain-Expert Comments

  • Domain-expert comments are appended before planning to correct minor issues or clarify ideas, improving metrics and conclusions while sometimes determining whether useful results are found.Examples include replacing direct string matching with an LLM-as-a-judge metric and using normalized partial-progress scores when task completion is usually zero.Source: D Example Domain-Expert Comments · p. 15

E Example Idea, Comments, and Plan

F Prompts

  • The core prompts instruct the system to generate new research ideas from supplied papers and available codeblocks, including novel or incremental follow-on ideas.The prompt encourages filling gaps, abstracting, combining, extending, and challenging assumptions in prior research programs.Source: F Prompts · p. 18, Ideation Prompt · p. 19

Ideation Prompt

  • The ideation prompt requires five structured ideas specifying hypotheses, variables, metrics, baselines, pilot tests, experiment designs, codeblocks, and exhaustive resource requirements.It emphasizes detailed implementation, data, outputs, evaluation, reporting, and feasibility documentation rather than high-level descriptions.Source: Ideation Prompt · pp. 20–21
  • Ideation Prompt requires exhaustive code-and-resource descriptions so the experiment workspace can assess feasibility before committing substantial time or money.Ideas should emphasize existing codeblocks and minimize external packages because unavailable or unfamiliar dependencies can hinder implementation.Source: Ideation Prompt · p. 22

Planning Prompt

  • Planning Prompt converts high-level research ideas into detailed, codeblock-conditioned experiment instructions while specifying baselines, comparisons, and required JSON outputs.The experiment builder should use available templates to reduce implementation errors, and codeblock names must match exactly or the experiment may fail.Source: Planning Prompt · pp. 23–24
  • Planning Prompt mandates MINI_PILOT, PILOT, and FULL EXPERIMENT scales, with progressively larger data or episode subsets and final evaluation on the test set.The system should run MINI_PILOT first, then PILOT, and stop before FULL EXPERIMENT for human verification; pilot levels differ in scale, not quality.Source: Planning Prompt · p. 24

Experiment Debugging Prompt

  • Experiment Debugging Prompt asks ScientistGPT to inspect prior instructions, generated code, dependencies, and execution results, then fix errors or approve the experiment.The reflection step uses logs and other run outputs to determine whether the code and results satisfy the specified requirements.Source: Experiment Debugging Prompt · p. 25
  • The debugging workflow requires scientifically rigorous, testable code with thorough error checks, informative logs, and reflection-based correction of actual implementation problems.It specifically requires verifying LLM response formats and parsers, detecting agent-mode failures, and avoiding hallucinated errors.Source: Experiment Debugging Prompt · pp. 26–27
  • Experiments use three scale-based pilot modes: MINI_PILOT for rapid small-subset checks, PILOT for moderate debugging and verification, and FULL EXPERIMENT for complete train-development-test evaluation.Pilot levels differ in scale rather than quality; the full experiment uses all data and steps, tunes hyperparameters on development data, and evaluates on the test set.Source: Experiment Debugging Prompt · pp. 26–27
  • The procedure prioritizes faithful execution through human-vetted codeblocks and common-library functions, while allowing data-volume reductions to meet the 7200-second per-iteration container limit.The prompts prohibit simulated procedures and recommend using known-good implementations to reduce errors, costs, and false positives or negatives.Source: Experiment Debugging Prompt · p. 27

Reporting Prompt

  • The reporting prompt directs the system to produce a short, faithful scientific paper from the task, code, results, saved files, and logs, covering the hypothesis, testing, findings, limitations, and fidelity.The output is complete LaTeX code intended for direct compilation and may include tables, figures, or other scientific content when needed.Source: Reporting Prompt · p. 29
  • The Reporting Prompt instructs ScientistGPT to produce a complete LaTeX scientific report containing appropriate tables, figures, analyses, and other supporting content.The report must be emitted as complete LaTeX within a single code block.Source: Reporting Prompt · p. 29

Experiment Summary Prompt (per experiment)

  • The Experiment Summary Prompt asks the system to interpret each experiment’s hypothesis, implementation fidelity, results, limitations, and conclusion using structured summary and classification fields.Inputs include the task description, generated code and requirements, results file, and optionally a log; outputs classify hypothesis status as support, reject, or inconclusive and fidelity as faithful, deviations, or errors.Source: Experiment Summary Prompt (per… · p. 30

Meta-Analysis Prompt

  • The Meta-Analysis Prompt aggregates differently implemented experiments around one idea, assigning support, refute, or inconclusive judgments and counts before generating an overall summary.It requires experiment-level reasoning, structured counts, and a detailed natural-language synthesis grounded in the supplied results, with explicit instructions not to hallucinate.Source: Meta-Analysis Prompt · p. 32

G Explanations of Incremental Novelty Claims … • Weak overall correlation (mean r = 0.16 across episodes)

The paper reports incremental-novelty assessments for six candidate discoveries and presents a pilot study of LLM confidence in TextWorldExpress CookingWorld. Across 50 episodes and 642 state predictions, confidence showed weak and highly variable alignment with prediction accuracy, with only slightly-better-than-random discrimination.

G Explanations of Incremental Novelty Claims

H Experiment Reports

H.1 Report: State Prediction Confidence

Report: State Prediction Confidence (Page 2)

  • For each step, the procedure recorded the current state and action, obtained a confidence-scored prediction, compared it with the actual next state, and scored accuracy using an LLM judge.Confidence scores ranged from 0–100, and GPT-4-mini generated predictions and accuracy scores.Source: Report: State Prediction Confidence… · p. 36

3.1 Data Overview

  • The experiment collected 642 state predictions across 50 episodes in TextWorldExpress’s CookingWorld.Episodes used simple three-room layouts with two ingredients and two distractor items, and lasted up to 25 steps.Source: §2 · p. 35, §3.1 · p. 36

• Weak overall correlation (mean r = 0.16 across episodes)

4 Discussion … • Each increase in complexity led to a decrease in accuracy

This section discusses LLM simulation accuracy in the CookingWorld interactive environment, examining the effects of state-representation complexity and the limits of confidence-based prediction. It also addresses constraints on generalizability, scoring validity, model coverage, and implementation completeness.

4.2 Limitations

• Use of LLM-as-judge for accuracy scoring may introduce bias

4.3 Implementation Fidelity

4.1 Key Findings

  • LLM confidence scores may not reliably indicate prediction accuracy in interactive environments.The conclusion recommends more sophisticated confidence estimation and testing across environments and LLM architectures.Source: §4.1 · p. 37, §5 · p. 38

Code Listing: State Prediction Confidence

  • The generated experiment recorded episode-level confidence and accuracy values and calculated their correlation before logging the result.The code also generated visualizations and saved experiment results.Source: Code Listing: State Prediction… · p. 42

4.1 Key Findings

  • 94.5% accuracy was achieved with boolean states, while full state representations reached 81.9%, showing decreasing simulation accuracy as state complexity increased.The experiment compared boolean, numerical, relational, and full representations in CookingWorld.Source: §H.2 · p. 43, §4.1 · p. 45

1 Introduction

  • The study examined how state-representation complexity affects LLM prediction of action-driven state transitions in the CookingWorld environment.It evaluated four complexity levels across 25 episodes with up to 25 steps each.Source: §1 · p. 43, §H.2 · p. 43

2.1 Experimental Design

  • The experiment implemented four levels of state representation complexity and ran in PILOT mode, including a full representation with dynamics and complete textual descriptions.The supplied passages identify the experimental design and the full-state condition but do not provide further implementation details.Source: §2.1 · p. 43, • Full: Complete state including… · p. 44

Report: Progressive State Complexity (Page 4) · 5 Conclusion · Code Listing: Progressive State Complexity

This section examines progressive state-complexity experiments by evaluating LLM-predicted versus actual states across complexity levels and statistically analyzing the resulting accuracy differences. It also describes the implementation, visualizations, and saved results, while noting the study’s stated experimental limitations.

Report: Progressive State Complexity (Page 4)

  • 6.5 percentage points was the largest accuracy drop, occurring between boolean and numerical representations; full state achieved the lowest accuracy at 81.9% with highest variance.The conclusion reports that simple boolean states can exceed 90% accuracy, while performance degrades as complexity increases.Source: Report: Progressive State Complexity… · p. 46, §5 · p. 46

4.2 Statistical Significance

  • Differences between complexity levels were statistically significant for every pairwise comparison except numerical versus relational levels (p = 0.819).Bootstrap resampling yielded p ¡ 0.001 for the other pairwise comparisons.Source: §4.2 · p. 46

4.3 Limitations

  • The experiment used random rather than goal-directed actions, was specific to CookingWorld, used gpt-4o-mini, and ran fewer PILOT episodes than the planned full experiment.These limitations constrain behavioral generalization, model-capability interpretation, and validation of the findings.Source: §4.3 · p. 46

5 Conclusion

  • The study concludes that state-representation design materially affects LLM simulation accuracy and recommends larger models, more episodes, and goal-directed behavior for future validation.The experiment followed PILOT specifications with logging, error handling, and statistical analysis.Source: §5 · p. 46

Code Listing: Progressive State Complexity

  • The implementation progressively extracts boolean, numerical, relational, or full states, prompts gpt-4o-mini for JSON next-state predictions, and scores field-level agreement against actual states.Episodes select random valid actions, while validation enforces the expected fields for each complexity level and comparison computes correct predictions divided by total comparisons.Source: Code Listing: Progressive State… · pp. 47–50
  • For each episode step, the implementation compares predicted and actual next states at the selected complexity level and records the resulting accuracy.It logs the action, predicted state, actual state, and accuracy before returning episode-level accuracy values.Source: Code Listing: Progressive State… · p. 50
  • The experiment runs cookingworld episodes across configured complexity levels and training seeds, retaining valid per-episode accuracies for each level.It initializes TextWorldExpressEnv, loads the cookingworld game, and aggregates episode accuracies by complexity level.Source: Code Listing: Progressive State… · p. 50
  • The analysis computes mean accuracy per complexity level and performs pairwise bootstrap statistical comparisons, treating the more complex state as experimental and the simpler state as baseline.Comparisons use equal-length paired arrays when levels contain data and store the resulting statistics in the analysis output.Source: Code Listing: Progressive State… · pp. 50–51
  • The pipeline generates an accuracy-by-complexity plot when data exist and saves the analysis, including raw accuracies and statistical tests, to results.json.The plot is written to to_save/accuracy_by_complexity.pdf, while missing plotting data triggers a warning and early return.Source: Code Listing: Progressive State… · p. 51

H.3 Report: Graph Alignment Metric … 4. Generated baseline predictions (random and constant)

This section documents CodeScientist experiments on graph-alignment scoring and single-stage versus two-stage text-based game generation, including generated reports, visualizations, code, and evaluations of execution, mechanics completeness, generation time, and action-outcome prediction.

1 Automatically Generated Summary

  • CodeScientist’s automatically generated summary is presented in Table 1, alongside a progress-correlation figure and example graphs ranging from simpler to most complex.The figures include examples from Game 13, Episode 8, Step 15 and Game 10, Episode 0, Step 0.Source: §1 · p. 53, Table 1 · p. 54, Figure 1 · p. 55

Code Listing: Graph Alignment Metric

  • The experiment code converts TextWorld cooking-environment observations into directed graphs containing extracted objects, relationships, and valid-action edges.Objects become typed nodes, textual relationships become relation edges, and supported actions create player-to-object action edges.Source: Code Listing: Graph Alignment Metric · p. 57
  • Similarity is evaluated with word-overlap, Jaccard, and custom scores based on graph nodes, spatial relations, and action relations.The custom score weights nodes, relations, and actions at 0.5, 0.3, and 0.2, respectively.Source: Code Listing: Graph Alignment Metric · pp. 57–58

2 Methodology

3 Results

  • 96.7% versus 66.7% mechanics completion favored two-stage over single-stage generation, while both achieved 100% execution success.Single-stage generation often omitted proper win conditions despite implementing other mechanics.Source: §3 · p. 62, Report: Multi-Stage Environment… · p. 63

Report: Multi-Stage Environment Generation (Page 3)

4 Discussion

  • The results support decomposing complex game generation into focused subtasks because staging basic mechanics before scoring and win conditions improved consistency and completeness.The benefit was framed as most useful when correctness is prioritized over generation speed.Source: §4 · p. 64

5 Limitations

  • The findings are limited by the simple game format, evaluation of mechanic presence rather than correctness or gameplay quality, use of one model, and a 120-generation sample.Results may differ for more complex games, and the sample may not capture all generation patterns.Source: §5 · p. 64, • Only one model (gpt-4o-mini) was… · p. 64

Code Listing: Multi-Stage Environment Generation

  • The generated experiment code evaluated syntax and required mechanics, stored game states and scores, computed word-overlap, Jaccard, and custom similarities, and applied bootstrap statistical analysis.It also generated similarity distributions and similarity-versus-progress plots.Source: Code Listing: Graph Alignment Metric · p. 60, Code Listing: Multi-Stage Environment… · p. 66
  • The generated experiment compares single-stage and two-stage game generation using execution success, syntax errors, mechanics completion, and generation time.The code runs both methods across configured games and generations, records evaluation metrics, computes summary statistics, and produces comparison plots.Source: Code Listing: Multi-Stage Environment… · pp. 67–68

H.5 Report: Simulation Confidence

  • GPT-4o-mini reached 65.7% action-prediction accuracy versus a 50% random baseline and showed a moderate confidence–accuracy correlation of r = 0.335.The pilot covered 50 games and 496 total action predictions; both results were statistically significant at p < 0.001.Source: §H.5 · p. 69

1 Introduction

  • The simulation-confidence study uses a controlled text-based cooking-game environment to test whether language models can predict action outcomes and provide meaningful confidence scores.The experiment collected data from 50 games with 10 actions per game, yielding 496 total predictions.Source: §1 · p. 69, Report: Simulation Confidence (Page 2) · p. 70

3 Results … • Protein measurements collected

These results evaluate agent performance across prediction and scientific-discovery tasks, examining prediction accuracy and confidence calibration alongside process scores and task completion relative to a baseline.

3.1 Prediction Accuracy

  • 65.7% overall accuracy across 496 predictions significantly exceeded the 50% random baseline (p ¡ 0.001).Individual game accuracies ranged from 20% to 100%.Source: §3.1 · p. 70

3.2 Confidence Analysis

  • Confidence correlated positively with prediction accuracy (Pearson’s r = 0.335, p ¡ 0.001), with higher average confidence for correct predictions.Figure 2 reports the correct-versus-incorrect confidence comparison, indicating some calibration in uncertainty estimates.Source: Report: Simulation Confidence (Page 3) · p. 71, Figure 2 · p. 71

• Data storage and visualization

5 Conclusion

  • Performance varied substantially across game contexts, suggesting that LLM reasoning consistency in interactive environments remains an open challenge.The conclusion characterizes the predictions as effective overall while emphasizing the need to understand and improve cross-context consistency.Source: §3.1 · p. 70, §5 · p. 72

Code Listing: Simulation Confidence

  • Generated code analyzed confidence against prediction accuracy, separated confidence values for correct and incorrect predictions, and compared experimental outcomes with a 0.5 baseline.The code also logged LLM accuracy, confidence–accuracy correlation, and bootstrap statistical-analysis results.Source: Code Listing: Simulation Confidence · p. 76

H.6 Report: Graph Agent for Discovery

  • Process scores were higher for the knowledge graph agent than the ReAct baseline (mean=0.29 vs 0.12, p¡0.001), but neither agent completed the task objectives.The result comes from a pilot study with 50 episodes in a proteomics investigation task.Source: §H.6 · p. 77

2.2 Task Description

1 Introduction

  • The experiment tested whether structured knowledge representation improves exploration and hypothesis generation for scientific discovery.The knowledge graph memory system was compared with a standard reactive agent.Source: §1 · p. 77

2.1 Experimental Design

  • The knowledge graph agent tracked objects, properties, measurements, and hypotheses in a DOT-format graph.The baseline ReAct agent used basic state tracking.Source: §2.1 · p. 77

Report: Graph Agent for Discovery (Page 2)

2.3 Metrics

  • The report specified primary evaluation metrics for assessing the agents.Source: §2.3 · p. 78

3 Results … • Computation time

This section presents experimental implementations and evaluations of agents and algorithmic approaches across protein-outlier discovery and resistor-substitution tasks, including performance comparisons, graph analyses, statistical tests, and identified limitations.

3.1 Performance Comparison

  • Agent performance differed significantly, with bootstrap analysis confirming a statistically significant difference in process scores (p < 0.001).Source: §3.1 · p. 78

3.2 Knowledge Graph Analysis

4.2 Limitations

  • Several limitations were identified, although the supplied passage does not specify their content.Source: §4.2 · p. 80

4.3 Implementation Fidelity

  • The implementation partially met the original specifications, with the supplied passage identifying an achieved component without further detail.Source: §4.3 · p. 80

Code Listing: Graph Agent for Discovery

  • The graph agent measures animal protein levels, identifies outliers using a two-standard-deviation z-score threshold, and records hypotheses and supporting evidence in a knowledge graph.The implementation tracks protein A and B values, generates animal-is-outlier hypotheses, and adds protein-specific evidence nodes and edges.Source: Code Listing: Graph Agent for Discovery · p. 83
  • The knowledge-graph agent prioritizes acquiring a proteomics meter, measuring unmeasured animals, analyzing protein-level outliers, and exploring when no higher-priority action remains.Its control logic ends when outliers are found and all accessible animals are measured.Source: Code Listing: Graph Agent for Discovery · pp. 84–85
  • The agent maintains a knowledge graph by extracting objects and properties, tracking locations and inventory, and storing measurements with protein_a and protein_b values.Successful measurements create measurement nodes and connect them to protein-property nodes.Source: Code Listing: Graph Agent for Discovery · pp. 85–86
  • The experiment runner compares knowledge-graph and baseline agents over seeds and difficulty levels, recording completion, success, normalized process score, steps, protein levels, nodes, edges, and hypotheses.The code separately plots process scores and knowledge-graph complexity, then applies statistical analysis with bootstrap resampling.Source: Code Listing: Graph Agent for Discovery · pp. 86–87

H.7 Report: Combinatorial Optimization

  • 100% accuracy within 1% tolerance was achieved by mathematical optimization, compared with 94% for the simple baseline and 24% for the LLM approach.These results came from testing 50 random target resistances between 10Ω and 1MΩ.Source: §H.7 · p. 89

1 Introduction

  • The study compared GPT-4 prompting, nearest single values or basic series combinations, and exhaustive series/parallel search using standard E24 resistor values.The implicit hypothesis was that an LLM could suggest effective combinations by understanding domain concepts from training.Source: §1 · p. 89

2 Methodology

  • The experiment evaluated 50 random resistance targets from 10Ω to 1MΩ, with three trials per target and success rates measured at 1%, 5%, and 10% tolerances.All three methods generated combinations from standard E24 series values.Source: §2 · p. 90, • Percentage error from target value · p. 90

• Percentage error from target value

3 Results … Code Listing: Combinatorial Optimization

This section presents an experiment comparing mathematical optimization, a simple baseline, and an LLM approach for resistor-combination optimization. It describes the evaluation configurations, metrics, significance analyses, implementation, and practical limitations.

3.1 Accuracy

  • 100% of mathematical-optimization suggestions were within 1% of target values, compared with 94% for the simple baseline and 24% for the LLM approach.These results establish mathematical optimization as the most accurate method for the task.Source: §3.1 · p. 90

3.2 Computation Time

  • The simple baseline was fastest at 0.004s mean, followed by the LLM approach at 1.157s and mathematical optimization at 3.226s.The report characterizes mathematical optimization’s accuracy advantage as potentially justifying its longer computation time.Source: §3.2 · p. 90

3.3 Error Distribution

4 Discussion

  • The results reject the hypothesis that the LLM approach can compete with mathematical methods because resistor optimization has well-defined mathematical rules and constraints.This interpretation directly links the observed performance gap to the task’s mathematical structure.Source: §4 · p. 91

5 Limitations

  • The study did not consider real-world component cost or availability, and mathematical optimization’s longer runtime may be prohibitive for some applications.These limitations qualify the practical conclusions despite mathematical optimization’s accuracy advantage.Source: Report: Combinatorial Optimization… · p. 92

6 Conclusion

  • Traditional algorithms significantly outperform current LLM capabilities for resistor substitution; the simple baseline balances speed and accuracy, while mathematical optimization maximizes accuracy at higher computational cost.Future work could test improved prompting or different LLM models, although the report expects traditional algorithms to remain superior.Source: §6 · p. 92

Code Listing: Combinatorial Optimization

  • The implementation generates resistor values across multiplier ranges, evaluates series and parallel combinations, and selects the lowest-error configuration through exhaustive search.The mathematical baseline enumerates component counts, combinations with replacement, and connection patterns.Source: Code Listing: Combinatorial Optimization · pp. 93–94
  • The LLM implementation prompts for JSON resistor combinations, extracts and validates codeblocks, checks required fields and connection counts, then computes actual resistance.The prompt provides series and parallel formulas, magnitude guidance, and example combinations.Source: Code Listing: Combinatorial Optimization · pp. 94–95
  • Bootstrap difference-score analysis compares experimental and baseline methods, interpreting p < 0.05 as a significant difference when valid comparison data exist.The code logs warnings when no valid comparison data are available and errors when statistical analysis fails.Source: Code Listing: Combinatorial Optimization · p. 96
  • Pilot, follow-on, and full experiments use progressively larger target sets and repeated runs, with up to 100 values, three resistors, and five runs per value.The follow-on configuration uses 50 values and the full experiment uses 100 values, both with full E24 resistor values.Source: Code Listing: Combinatorial Optimization · pp. 96–97
  • The experiment compares LLM, Simple, and Mathematical methods for approximating randomly generated resistance targets between 10Ω and 1MΩ.Targets are generated with a fixed NumPy seed for reproducibility.Source: Code Listing: Combinatorial Optimization · p. 97
  • For each method and target, the code records percent error, within-1%, within-5%, and within-10% success indicators, component count, and computation time.Failed suggestions receive infinite percent error and false success indicators.Source: Code Listing: Combinatorial Optimization · p. 97
  • The analysis summarizes mean and median percent error, threshold success rates, and mean computation time for each method.Results are saved, plotted, statistically analyzed, and logged by method.Source: Code Listing: Combinatorial Optimization · p. 98

Related papers

Loading 2503.22708v1…