PaperlayerView exact revision on arXiv

Source-linked AI summary

CodeScientist: End-to-End Semi-Automated Scientific Discovery with Code-based Experimentation

Peter Jansen, Oyvind Tafjord, Marissa Radensky, Pao Siangliulue, Tom Hope, Bhavana Dalvi Mishra, Bodhisattwa Prasad Majumder, Daniel S. Weld, Peter Clark

arXiv:2503.22708v1cs.AIcs.CL

TL;DR

Autonomous scientific discovery systems often explore constrained code variants and evaluate generated artifacts mainly through paper review. CodeScientist jointly searches literature and codeblocks to generate and test ideas, yielding 19 candidate discoveries, 6 of which met minimum soundness and incremental-novelty thresholds after external review and code review.

  • Problem

    Existing autonomous scientific discovery systems explore constrained design spaces and often evaluate generated artifacts with limited code review.

  • Method

    CodeScientist uses genetic search over combinations of research literature and domain codeblocks to generate, execute, and analyze experiments.

  • Results

    19 candidate discoveries were identified, with 6 meeting minimum thresholds for scientific soundness and incremental novelty after external review and code review.

  • Takeaways & Limitations

    The candidate discoveries broaden the explored scope beyond benchmark optimization to include new tasks, agents, metrics, and data.

  • Takeaways & Limitations

    Validating candidate discoveries requires intensive code examination and rerunning experiments because paper review alone may not assess faithfulness.

Abstract

from arXiv · show

Despite the surge of interest in autonomous scientific discovery (ASD) of software artifacts (e.g., improved ML algorithms), current ASD systems face two key limitations: (1) they largely explore variants of existing codebases or similarly constrained design spaces, and (2) they produce large volumes of research artifacts (such as automatically generated papers and code) that are typically evaluated using conference-style paper review with limited evaluation of code. In this work we introduce CodeScientist, a novel ASD system that frames ideation and experiment construction as a form of genetic search jointly over combinations of research articles and codeblocks defining common actions in a domain (like prompting a language model). We use this paradigm to conduct hundreds of automated experiments on machine-generated ideas broadly in the domain of agents and virtual environments, with the system returning 19 discoveries, 6 of which were judged as being both at least minimally sound and incrementally novel after a multi-faceted evaluation beyond that typically conducted in prior work, including external (conference-style) review, code review, and replication attempts. Moreover, the discoveries span new tasks, agents, metrics, and data, suggesting a qualitative shift from benchmark optimization to broader discoveries.

1 Introduction

CodeScientist extends automated scientific discovery beyond narrowly constrained search spaces by combining literature and code through genetic search. In experiments across agents and virtual environments, it produced 19 suggested discoveries, 6 of which appeared to meet minimum thresholds, while motivating more rigorous evaluation methods.

  • Motivation: Prior automated scientific discovery systems have succeeded in targeted domains but often rely on custom systems and large hand-crafted search spaces.Recent language-model-based systems aim to cover more of the research pipeline, while simplifying the problem through restrictions such as prewritten-code variants, domain-specific languages, or constrained domains.
  • System contribution: CodeScientist applies genetic search over combinations of research literature and code to support ideation and experiment execution.The system uses papers of interest, short example codeblocks, and reusable actions such as language-model calls.
  • Experimental scope: 19 suggested discoveries emerged from hundreds of experiments in the broad domain of agents and virtual environments.The experiments were conducted at scale rather than in a single narrowly defined benchmark setting.
  • Evaluation: 6 of the 19 discoveries appeared to meet minimum thresholds after evaluation beyond conventional paper review.The introduction frames discovery evaluation as difficult because gold-annotated outcomes are generally unavailable, motivating alternative assessments such as rigorous manual review.

Ideation

CodeScientist generates ideas by mutating combinations of papers and codeblocks, then uses human selection and comments to refine promising proposals. One proposed study tested whether self-assessed LLM confidence correlates with accuracy in state prediction.

  • Ideation: The system uses an LLM as a mutator that takes papers and codeblocks as input and outputs research ideas.This defines the core ideation mechanism.
  • Ideation: Idea #314 hypothesized that self-assessed LLM confidence correlates with accuracy in state prediction tasks.The pilot used TextWorldExpress to generate 200 state-prediction pairs, focusing on boolean property predictions with conference scores.
  • Ideation: Humans select a subset of promising ideas for further consideration.Selection occurs after batch idea generation.
  • Ideation: Humans provide brief comments on each selected idea to guide refinement.The process includes explicit human feedback on proposed ideas.
  • Ideation: Comments on Idea #314 recommended using an LLM-as-a-judge metric instead of raw string matching for greater robustness to presentation variations.The recommendation specifically changes how predictions are evaluated.

Planning

The planning workflow combines ideation, code-based experiment construction and execution, result reporting, and cross-experiment analysis. A representative plan specifies the environment, model, data collection, pilot and full runs, and subsequent human verification.

  • Experiment plan: Batch Idea #314 uses TextWorldExpress CookingWorld environments with 3 ingredients, 2 distractors, and no doors, while calling gpt-4o-mini.The plan prompts the model for state predictions with confidence scores and uses an LLM judge for accuracy scoring from 0 to 1.
  • Experiment plan: The experiment includes a pilot of 3 episodes with 10 steps each and a full run of 50 episodes with 25 steps each.The planned implementation iterates over episodes, evaluates each episode, and records accuracy versus confidence.
  • Evaluation: A human examines and verifies the code and results after the automated experiments.The workflow also reports that candidate discoveries undergo external review, replication, and code review.
  • Workflow: The workflow progresses through ideation, planning, code-based experimentation, reporting, and meta-analysis across experiments.Figure 1 identifies these as the core stages of the CODESCIENTIST discovery workflow.

2 Related Work

Recent autonomous scientific discovery systems vary in how they generate ideas, construct and execute experiments, and define their domains, while evaluation remains limited by the cost and incompleteness of artifact assessment. In particular, paper-only conference-style review may fail to detect whether reported discoveries faithfully match their implemented code.

  • Ideating and Executing Research: ASD systems primarily differ in ideation, experiment construction, execution, and problem domain.Examples span chemistry, biology, and AI, with AI systems summarized in Table 1.
  • Ideating and Executing Research: Unrestricted literature-based ideation can produce broadly scoped ideas that are challenging to implement.Ideating on literature is described as a common and naturalistic method.
  • Evaluating Discoveries: Automatic discovery evaluation is feasible only for limited domains and tasks with direct instrumentation or measurable objectives.Examples include molecular simulators, science-themed games, and benchmark performance improvements.
  • Evaluating Discoveries: Manual evaluation of research artifacts such as code is costly, while conference-style Likert review of papers alone may not fully assess discovery faithfulness.Reported paper claims can strongly differ from what the system actually implements in code.

3 System Overview

CodeScientist uses a five-step, largely prompt-driven workflow that combines curated papers and reusable codeblocks to generate, build, run, report, and analyze experiments. Its experiment builder iteratively generates and debugs code in an instrumented sandbox, while reporting summarizes outcomes for high-throughput review.

  • Workflow: The workflow has five major steps: ideation, planning, experiment building, reporting, and meta-analysis, implemented primarily as language-model prompts.Example prompts and additional implementation details are provided in Appendix F.
  • Inputs: CodeScientist requires a human-curated list of domain papers and relevant codeblocks for common tasks such as calling an LLM, implementing a ReAct agent, and loading benchmarks.The agent-centered corpus used in this work is described in Section 4.
  • Ideation: The ideator generates candidate ideas from recent research using structured hypothesis, variable, metric, baseline, pilot-design, and resource slots, plus crossover and mutation operators.A human selects an interesting subset of generated ideas and may add brief expert comments.
  • Experiment building: The experiment builder converts a plan and codeblocks into code, results, and logs through iterative generate-execute-reflect debugging until completion or a hard limit.Its components include initial Python code generation, an instrumented sandbox that captures outputs and tracks API usage, and reflection-based code modification.
  • Reporting: Completed experiments produce detailed LaTeX reports and short summaries that identify whether hypotheses were confirmed, rejected, or inconclusive.The summaries help users prioritize full reports when experimental throughput exceeds human reading capacity.
  • Repeated execution: The system’s variability motivates repeated runs, because the builder can produce different implementations even with the same plan, codeblocks, and low generation temperature.This variation arises from language-model variability and autoregressive conditioning on previous debug-cycle outputs.

4 Discovery Experiments

CodeScientist assembled a corpus of papers and reusable code snippets, screened approximately 2,000 generated ideas to select 50, and ran 250 experiments. It flagged 19 ideas for human inspection using external review and internal code, log, and replication checks.

  • Experiment Setup: 57 papers and 10 code snippets formed the corpus for experiments on agent architectures and virtual environments.The snippets covered language-model calls, ReAct agents, plotting, inferential statistics, knowledge graphs, and benchmark environments.
  • Idea Selection: Approximately 2000 candidate ideas from 200 randomly selected paper combinations were screened until a domain expert selected 50 viable, sufficiently different ideas.The expert also provided brief comments on each selected idea before plans were generated.
  • Experiment Builder: 250 experiment runs resulted from giving each of the 50 ideas five attempts to generate functioning code and results.Cost, runtime limits, and actual usage statistics were reported separately in Table 3.
  • Candidate Discoveries: 19 of 50 ideas were flagged for human inspection because at least one of five runs produced “interesting” results.These candidate discoveries were then evaluated through external and internal review.
  • External Review: 3 external research scientists rated generated papers for soundness and novelty using a conference-style review rubric.Ratings were converted to binary scores, with unsound or non-novel experiments treated as failures and minimally sound or incrementally novel results meeting the threshold.
  • Internal Review: 1 domain expert reviewed code and experiment logs and attempted replication with more samples, retaining veto power over discoveries that failed detailed examination.This internal review could reject results that appeared genuine from the paper and external review.

Human Reviewers

Human evaluation found that 6 of 19 candidate discoveries met minimum soundness and incremental novelty criteria after external review, code and experiment-log inspection, and replication attempts. The surviving discoveries included new tasks, benchmarks, metrics, methods, and challenges to prevailing assumptions, while rejected results commonly reflected weak baselines, sampling effects, or implementation errors.

  • Candidate discoveries: 6 of 19 discoveries passed both external and internal review after human evaluation of papers, code, experiment logs, and replication attempts.13 of 19 met minimum soundness and novelty criteria with at least 2 of 3 external reviewers, but internal review rejected 7 of those 13.
  • Candidate discoveries: The passing discoveries spanned new tasks, benchmarks, metrics, methods, and challenges to assumptions, rather than only benchmark performance improvements.The paper states that most surviving discoveries involved creating or questioning these broader forms of scientific contribution.
  • Candidate discoveries: One surviving discovery found that an LLM’s self-assessed confidence had low correlation with actual prediction accuracy in state-prediction tasks.The state-prediction data was automatically crawled from a benchmark, and the correlation remained consistently low across experiments despite variation.
  • Rejected discoveries: 13 of 19 discoveries were rejected after human evaluation, including results with overly simplistic baselines or models, effects that disappeared with more samples, and major implementation errors.The reported failure modes included 6 discoveries with overly simplistic baselines or models, 3 whose effects disappeared after rerunning with larger samples, and 2 with major implementation errors.

5 Discussion

The discussion identifies limitations in idea diversity, experiment implementation, result faithfulness, and research-method adherence, while outlining human involvement and fully automated operation as ongoing challenges. In a pilot, autonomous execution of 100 ideas appeared to produce 2 candidate discoveries, but high variance, low discovery rates, and prohibitive cost limited inference.

  • Idea Diversity: Most generated ideas are highly similar or mechanical variations, despite candidate discoveries spanning new agents, tasks, metrics, methods, and benchmarks.The authors relate this saturation to prior observations about LLM ideators and discuss uniqueness filtering as a targeted improvement.
  • Experiment Builder Failures: 59% of planner-designed experiments cannot be successfully implemented, with failures commonly reaching debugging limits or experiment time limits.Among executor errors, 32% hit the maximum number of debug iterations and 18% hit experiment time limits.
  • Unfaithful Experiments: Human inspection found an apparent graph-agent improvement was false because the agent built but never used the graph, instead selecting actions randomly.Detecting such unfaithful experiments is difficult and laborious, motivating automated methods to reduce false positives and negatives.
  • Adherence to best research practices: Generated experiments often violate research best practices, including training-set evaluation, statistical errors, and weak baselines that can create spurious significance.One example compared modified REACT agents against random baselines rather than REACT baselines.
  • Reducing Human Effort: The system currently requires human involvement at 5 steps because fully automated pilots produced more duplication, debugging failures, and methodologically unconvincing results.Domain-expert comments before planning help address obvious methodological issues, while improved model knowledge may reduce this reliance.
  • Fully-automated Mode: 2 candidate discoveries emerged from 100 ideas autonomously generated and executed in a pilot, but high variance, low discovery rates, and prohibitive cost prevent statistically powered ablations.These constraints make fully automated operation possible but currently impractical for reliable large-scale inference.

6 Conclusion

CodeScientist is an end-to-end semi-automated discovery system that performs genetic search over combinations of literature and codeblocks before experimentally analyzing software artifacts. Applied to agents and environments, it identified 19 potential discoveries, 6 meeting minimum thresholds for scientific soundness and incremental novelty after external conference-style review.

  • CodeScientist performs genetic search jointly over combinations of literature and codeblocks before building, running, and analyzing software artifacts in experiments.
  • 19 potential discoveries were identified from literature and common codeblocks in the domain of agents and environments.
  • 6 discoveries met minimum thresholds for scientific soundness and incremental novelty after external conference-style review.

7 Limitations … B External Reviewer Rubric

CodeScientist’s candidate discoveries face limitations from low-budget experimentation, difficult code-focused validation, incremental novelty, and unresolved ideator recall. External reviewers assessed the papers using separate categorical scales for scientific soundness and novelty.

  • 7 Limitations: ≈$4 and approximately 2 hours are the average experiment’s cost and completion time, enabling rapid iteration but limiting sample sizes.The low sample counts can produce false positives and miss all but the largest effects, producing false negatives.
  • 7 Limitations: The six discoveries are framed as “candidate discoveries” because inaccuracies may remain undetected despite external and internal review.Reviewing long LLM-generated code and rerunning discoveries at greater scale is more effortful than typical peer review.
  • 7 Limitations: The six expert-validated discoveries would likely be categorized as normal incremental science rather than transformational discoveries.Ratings by the three external reviewers suggest that each candidate was, at best, incrementally novel.
  • 7 Limitations: Ideator recall remains an open research area because many runtime-generated ideas had issues, while pragmatic cost savings required filtering a large idea set.Reported issues included near duplicates, incorrect metrics, and ideas that were very challenging to implement.
  • B External Reviewer Rubric: External reviewers categorized each Table 4 paper on scientific soundness and novelty, provided rating justifications, and described its contributions and claims.Soundness concerns the rigor and reliability of methods and evidence supporting the claims.
  • B External Reviewer Rubric: The soundness rubric ranges from Clearly Sound and Likely Sound to Minor Concerns and Unsound.Minor Concerns allow methodological limitations that may slightly affect measurements without altering overall conclusions, whereas Unsound indicates flaws undermining credibility.
  • B External Reviewer Rubric: The novelty rubric distinguishes Highly Novel, two levels of Incrementally Novel variation, and Not Novel/Exists in Exact Form.The incremental categories cover substantial modifications with marked advancements and modest adaptations to established work.

C Meta-Analysis Categorization Criteria … Meta-Analysis Prompt

The paper defines meta-analysis categories by agreement and completion rates, while its prompts structure implementable research ideation and planning around papers, codeblocks, resources, and experiment specifications. Domain-expert comments refine ideas before planning, and the appendix provides examples and prompt templates supporting this workflow.

  • C Meta-Analysis Categorization Criteria: Consistent results require at least 80% (4 of 5) independent runs to support or reject the hypothesis in the same way.This criterion applies to a given generated idea.
  • C Meta-Analysis Categorization Criteria: Limited results occur when 40% or fewer (2 or fewer of 5) runs successfully complete, regardless of experimental outcome.Any remaining set of five experiments is classified as mixed if it is neither consistent nor limited.
  • D Example Domain-Expert Comments: Domain-expert comments are appended before planning to correct minor issues or clarify ideas, improving metrics and sometimes enabling stronger conclusions or results.Pilot experiments without comments generally still work, but may use less robust metrics or fail to find results.
  • Ideation Prompt: The ideation prompt asks ScientistGPT to generate 5 research ideas from papers, allowing highly novel or incremental ideas and encouraging gap-filling, abstraction, combination, extension, and assumption-challenging.Ideas are conditioned on available codeblocks and must include hypotheses, variables, metrics, baselines, pilots, designs, codeblocks, and required resources.
  • Ideation Prompt: Each proposed idea must specify an exhaustive list of required code, resources, and models because this information determines workspace preparation and experiment feasibility.The prompt warns that incomplete documentation can waste substantial time and money on infeasible ideas.
  • F Prompts: The prompt suite requires ideas to be implementable with available templates and encourages minimizing external libraries because the experiment builder may lack access to or fluency with them.The ideation output is formatted as JSON and emphasizes existing codeblocks in the experiment builder.
  • Planning Prompt: The planning prompt converts a paper-derived high-level idea into a specific prompt for a template-based experiment builder using existing codeblocks to reduce implementation and research-method errors.The planner is told that the supplied idea may contain inaccurate or unusable information and must be converted into detailed specifications.

G Explanations of Incremental Novelty Claims … • Weak overall correlation (mean r = 0.16 across episodes)

The paper assesses incremental novelty for six candidate discoveries and reports a pilot study testing whether LLM confidence predicts state-prediction accuracy in TextWorldExpress’s CookingWorld. Across 642 predictions from 50 episodes, confidence showed weak correlation with accuracy and only slightly exceeded random prediction performance.

  • G Explanations of Incremental Novelty Claims: Incremental novelty assessments were provided for the 6 candidate discoveries in Table 4.These assessments are presented in Table 8.
  • H.1 Report: State Prediction Confidence: The pilot study used TextWorldExpress’s CookingWorld to evaluate the relationship between LLM confidence scores and prediction accuracy.The experiment focused on state predictions in a text-based game environment.
  • 1 Introduction: The experiment examined whether LLM-generated confidence scores meaningfully correlate with actual prediction accuracy in a controlled game environment.The study used a text-based game setting to evaluate self-assessed prediction confidence.
  • 2 Methods: The data collection comprised 50 episodes of up to 25 steps each.The environment used simple 3-room layouts with 2 ingredients and 2 distractor items.
  • Report: State Prediction Confidence (Page 2): For each step, the procedure recorded the current state and action, obtained an LLM prediction with confidence scores (0-100), compared the prediction with the actual next state, and scored accuracy using LLM-as-judge.This sequence defined how confidence and prediction accuracy were paired.
  • 3 Results: 642 state predictions were collected across 50 episodes.Each prediction included a confidence assessment and was evaluated against the observed next state.
  • • Weak overall correlation (mean r = 0.16 across episodes): Correlation varied substantially between episodes, ranging from -0.04 to 0.57.The reported range indicates that the confidence-accuracy relationship was inconsistent across episodes.
  • Report: State Prediction Confidence (Page 3): ROC analysis found that confidence scores performed only slightly better than random at predicting accuracy, with AUC = 0.54.The overall confidence-accuracy correlation was weak, with mean r = 0.16 across episodes.

4 Discussion … • Each increase in complexity led to a decrease in accuracy

The study evaluated LLM state-transition prediction in CookingWorld across four representation-complexity levels and found that accuracy decreased as complexity increased. It also reports that LLM confidence was not strongly predictive of accuracy, while noting limitations in environment complexity, scoring, and model coverage.

  • 4.1 Key Findings: LLM confidence scores were not strongly predictive of actual prediction accuracy in this environment.The conclusion similarly characterizes confidence as potentially unreliable for prediction-quality assessment.
  • 4.2 Limitations: The experiment used a limited environment, LLM-as-judge accuracy scoring, and a single GPT-4-mini model, limiting generalization and potentially introducing bias.The stated limitations concern real-world complexity, scoring bias, and whether one model represents broader LLM capabilities.
  • 4.3 Implementation Fidelity: The implementation completed a pilot with 50 episodes, exceeding the requested 20, but bootstrap resampling for confidence intervals was not fully implemented.The study also reports that most key requirements were implemented.
  • 5 Conclusion: Future work should test more sophisticated confidence estimation and whether the findings generalize across environments and LLM architectures.The conclusion frames current confidence scores as potentially unreliable indicators of prediction accuracy in interactive environments.
  • Code Listing: State Prediction Confidence: The generated code collected predictions and confidence scores, scored accuracy with an LLM, calculated confidence-accuracy correlations, and produced scatter plots and ROC curves.The code used a CookingWorld environment, random valid actions, JSON-formatted responses, and logged experiment results.
  • Report: Progressive State Complexity (Page 2): 25 episodes with up to 25 steps each were used to measure prediction accuracy across the complexity levels.The study examined state-transition simulation in the CookingWorld environment.
  • 4 Discussion: Each increase in state complexity led to a decrease in prediction accuracy across the four tested representation levels.The levels were boolean, numerical, relational, and full.
  • 4.1 Key Findings: 94.5% accuracy was achieved by Boolean states, the highest level reported and evidence that simple binary predictions were easiest for the LLM.The experiment compared boolean, numerical, relational, and full state representations.

Report: Progressive State Complexity (Page 4) · 5 Conclusion · Code Listing: Progressive State Complexity

State representation complexity significantly affects LLM simulation accuracy: performance is high with simple boolean states but declines as representations become more complex. The study’s conclusions are qualified by domain, behavior, model, and episode-count limitations.

  • Report: Progressive State Complexity (Page 4): 6.5 percentage points marked the largest accuracy drop, occurring between boolean and numerical representations.
  • Report: Progressive State Complexity (Page 4): 81.9% was the lowest accuracy, observed for the full state representation, which also had the highest variance.
  • 4.2 Statistical Significance: p ¡ 0.001 indicated statistically significant differences for every pairwise complexity comparison except numerical versus relational levels, where p = 0.819.
  • 4.3 Limitations: The experiment used random rather than goal-directed actions, was specific to CookingWorld, and used gpt-4o-mini rather than testing larger LLMs.
  • 4.3 Limitations: The PILOT mode used fewer episodes than the originally specified FULL EXPERIMENT, limiting direct validation of the findings.
  • 5 Conclusion: The conclusion reports high accuracy (¿90%) with simple boolean states and substantial degradation as state representation complexity increases.
  • Code Listing: Progressive State Complexity: The generated code evaluates boolean, numerical, relational, and full states by extracting representations, prompting gpt-4o-mini for next-state JSON, and comparing predictions with actual states.

H.3 Report: Graph Alignment Metric … 4. Generated baseline predictions (random and constant)

The merged sections describe automated graph-based state representation and similarity scoring, then compare single-stage and two-stage text-game generation and evaluate GPT-4o-mini’s action predictions with confidence estimates. Two-stage generation improved mechanics completeness but increased runtime, while simulation predictions exceeded random accuracy and confidence correlated positively with correctness.

  • H.3 Report: Graph Alignment Metric: The report generator failed consistently on the graph-alignment experiment because it produced a very large number of figures, so an automated summary and selected figures were provided instead.The failure was attributed to generating one figure for each examined graph.
  • H.3 Report: Graph Alignment Metric: The graph-alignment experiment converts text-based game states into directed graphs containing extracted objects, relationships, and player-action edges.Objects become nodes, textual relationships become relation edges, and valid actions create player-to-object action edges.
  • H.3 Report: Graph Alignment Metric: The graph metric computes word-overlap, Jaccard, and custom similarity scores using graph nodes plus spatial and action relations.The custom score weights nodes at 0.5, relations at 0.3, and actions at 0.2.
  • 3 Results: Both single-stage and two-stage game generation achieved 100% execution success with no syntax errors, but differed in mechanics completeness and generation time.The experiment used 20 games with 3 generations per game per method, totaling 120 generations.
  • 3 Results: 96.7% mechanics completion was achieved by two-stage generation versus 66.7% for single-stage generation, largely because single-stage outputs often omitted win conditions.The two-stage process generated movement and inventory mechanics first, then added scoring and win conditions.
  • 4 Discussion: 29.8 seconds was the average two-stage generation time versus 16.9 seconds for single-stage generation, a 76% increase.The discussion characterizes this as a trade-off between improved completeness and additional generation overhead.
  • 5 Limitations: The two-stage approach supported more complete implementations, although the evaluation measured mechanics presence rather than correctness or gameplay quality and tested only gpt-4o-mini.The study used a relatively simple game format, so results may differ for more complex games.
  • H.5 Report: Simulation Confidence: 65.7% action-prediction accuracy exceeded the 50% random baseline, with p ¡ 0.001, and confidence correlated positively with accuracy at r = 0.335, p ¡ 0.001.The pilot covered 50 games and 496 total action predictions in a text-based cooking environment.

3 Results … • Protein measurements collected

The results show that the LLM predicted action outcomes above chance with partially calibrated confidence, while the broader graph-agent evaluation measured scientific-discovery performance through task completion, process score, graph complexity, and protein measurements.

  • 3.1 Prediction Accuracy: 65.7% overall accuracy across 496 predictions exceeded the 50% random baseline (p ¡ 0.001).Individual game accuracies ranged from 20% to 100% (mean = 65.7%, SD = 15.8%).
  • Report: Simulation Confidence (Page 3): Pearson’s r = 0.335 (p ¡ 0.001) showed a moderate positive correlation between confidence scores and prediction accuracy.Average confidence was higher for correct than incorrect predictions, indicating some calibration in uncertainty estimates.
  • 5 Conclusion: The study concluded that LLMs can predict action outcomes in text-based environments while providing meaningful confidence estimates.Performance varied substantially across game contexts, motivating further research on consistency in interactive-environment reasoning.
  • H.6 Report: Graph Agent for Discovery: The graph-agent study compared a knowledge graph agent with a baseline ReAct agent for scientific discovery in a proteomics investigation task.The knowledge graph agent tracked objects, properties, measurements, and hypotheses, whereas the baseline used reactive state tracking.
  • H.6 Report: Graph Agent for Discovery: The knowledge graph agent achieved significantly higher process scores than the baseline, with mean=0.29 vs 0.12, pThe supplied passage reports the comparison but truncates the p-value after “p”.
  • • Protein measurements collected: Primary evaluation metrics included task completion, process score, graph complexity, and protein measurements collected.The study assessed whether structured knowledge representation improved exploration and hypothesis generation over a standard reactive agent.

3 Results … • Computation time

The reported results found statistically significant differences in process scores between agents, while the implementation also identified limitations in task completion and hypothesis generation. A separate resistor-substitution evaluation found mathematical optimization outperforming both an LLM-based advisor and a simple baseline.

  • 3.1 Performance Comparison: Statistical analysis found significant differences between the agents, and bootstrap analysis confirmed a process-score difference at p ¡ 0.001.The reported comparison concerns agent performance and process scores.
  • 4.2 Limitations: The implementation partially met the original specifications, with limitations identified in task completion, hypothesis generation, measurement use, meter acquisition, and navigation.The listed limitations include no successful task completions, limited hypothesis generation, frequent failure to acquire the proteomics meter, and primarily random navigation.
  • Report: Graph Agent for Discovery (Page 4): The knowledge-graph code included pilot and full-experiment settings with configurable episode counts, step limits, seeds, and difficulty levels.The full-experiment configuration used 50 episodes per difficulty, 100 maximum steps, 50 seeds, and Easy, Normal, and Challenge difficulties.
  • Code Listing: Graph Agent for Discovery: The knowledge-graph implementation tracked measurements, constructed graphs, analyzed protein-level outliers with z-scores, and logged and visualized experimental results.The code defined graph nodes and edges, recorded protein measurements, analyzed deviations exceeding 2 standard deviations, and generated comparison plots.
  • Report: Graph Agent for Discovery (Page 3): The experiment ran separate knowledge-graph and baseline episodes, recorded completion, success, process score, steps, graph size, hypotheses, and protein levels, and applied bootstrap resampling.The code explicitly separated knowledge-graph and baseline results and stored the listed episode-level fields.
  • H.7 Report: Combinatorial Optimization: In resistor substitution, testing on 50 random targets showed 24% accuracy for the LLM approach, 94% for the simple baseline, and 100% for mathematical optimization within 1% tolerance.The targets ranged between 10Ω and 1MΩ, and the comparison used an LLM-based advisor, nearest-value baseline, and mathematical optimization.
  • 2 Methodology: The resistor experiment used 50 random resistance targets between 10Ω and 1MΩ, with 3 trials per target and standard E24 series values.Reported performance metrics included success rates at 1%, 5%, and 10% tolerances.
  • • Computation time: The resistor study evaluated approaches using success rates at 1%, 5%, and 10% tolerances, percentage error, number of components, and computation time.These metrics were listed as the evaluation criteria for the resistor-combination methods.

3 Results … Code Listing: Combinatorial Optimization

The mathematical approach was most accurate but slowest, while the simple baseline offered a faster compromise and the LLM approach performed poorly with high variance and failures. The study therefore favors traditional algorithms while noting limitations in evaluation scope, practical costs, and omitted real-world factors.

  • 3.1 Accuracy: 100% of mathematical suggestions were within 1% of target values, compared with 94% for the simple baseline and 24% for the LLM approach.The mathematical approach achieved the highest accuracy, while the LLM approach performed significantly worse.
  • 3.2 Computation Time: 0.004s was the simple baseline’s mean computation time, versus 1.157s for LLM and 3.226s for mathematical optimization.The mathematical approach’s higher accuracy may justify its longer computation time for most applications.
  • 3.3 Error Distribution: The mathematical and simple approaches maintained consistently low errors, whereas LLM errors had much higher variance and included complete failures.The error distributions show substantially less consistent LLM performance than the two algorithmic approaches.
  • 4 Discussion: The results reject the hypothesis that the LLM approach could compete with mathematical methods for resistor combination optimization.The discussion attributes this result to the task’s well-defined mathematical rules and constraints.
  • 4 Discussion: The LLM approach tended to suggest unnecessarily complex combinations, occasionally failed to generate valid solutions, and performed inconsistently across resistance ranges.These are identified as key limitations of the LLM approach.
  • 5 Limitations: 50 target values were tested, and only one LLM model/prompt combination was evaluated, limiting the study’s generalizability.The limitations section explicitly identifies the restricted target count and single model/prompt setup.
  • Report: Combinatorial Optimization (Page 4): Real-world component cost and availability were not considered, and mathematical optimization’s longer computation time may be prohibitive for some applications.These limitations constrain direct application of the reported results.
  • Code Listing: Combinatorial Optimization: The generated code evaluates LLM, simple, and mathematical methods using resistance-error metrics, tolerance success rates, plots, and bootstrap statistical analysis.The implementation tests targets across resistor combinations and records percent error, tolerance indicators, component counts, and computation time.
Loading 2503.22708v1…