PaperlayerView exact revision on arXiv

Source-linked AI summary

Can LLM Agents Discover? Evaluating Creativity on ML Engineering Tasks

Shitanshu Bhushan, Yunxiang Zhang, Lu Wang

arXiv:2608.30047v1cs.AI

TL;DR

The paper addresses whether autonomous LLM research agents can produce solutions that are both novel and useful. It evaluates creativity across multi-turn ML engineering trajectories using separate measures of psychological novelty, historical novelty, and usefulness. Agents explore highly novel solution regions, but their novelty declines during exploitation and does not translate reliably into improved task performance.

  • Problem

    Existing evidence does not establish whether autonomous LLM research agents produce solutions that are both novel and useful, especially across multi-turn trajectories.

  • Method

    The study evaluates P-Creativity, H-Creativity, and usefulness on 10 MLE-Bench competitions, using an LLM-as-a-Judge pipeline validated against human P-Creativity judgments.

  • Results

    Agents show greater H-Creativity than medal-winning humans, while P-Creativity declines as agents shift from exploration to exploitation and does not reliably track performance.

  • Takeaways & Limitations

    Current agents can explore novel solution regions, but novelty without usefulness remains insufficient for autonomous discovery.

  • Takeaways & Limitations

    The evaluation covers trajectories of up to 9 episodes; longer trajectories make full-history comparison computationally infeasible.

Abstract

from arXiv · show

Recent AI systems promise autonomous scientific discovery, claiming to discover algorithms and produce research papers, yet understanding whether they exhibit creativity, the capacity to produce solutions that are both novel and useful, remains an open question. We present a framework for evaluating multi-turn LLM research agents' creativity using ML engineering tasks as a testbed, through three dimensions: P-Creativity (psychological novelty: novel relative to the agent's own prior solutions within a run), H-Creativity (historical novelty: novel relative to the corpus of human solutions), and Usefulness (task performance). Evaluating two agent frameworks, AIDE and AIRA-Dojo, on 10 Kaggle-style machine learning tasks from MLE-Bench, we develop an LLM-as-a-Judge pipeline and verify its strong correlation with human creativity judgments, providing a reliable automated metric for P-Creativity evaluation at scale. Applying this pipeline to agent trajectories, we find: (1) all agents exhibit declining P-Creativity as they transition from exploration to exploitation; (2) LLMs exhibit greater H-Creativity than medal-winning humans, yet achieve lower performance. Our findings reveal that current agents can explore novel regions of the solution space but lack the capacity to convert this novelty into improved task performance.

1 Introduction

The paper asks whether multi-turn LLM research agents produce solutions that are both novel and useful, and introduces a framework to evaluate this at scale. Across ML engineering tasks, agents show novelty but struggle to convert it into improved performance.

  • The study addresses the lack of scalable measures that distinguish novelty relative to an agent’s experience from novelty relative to the human solution space.It evaluates creativity in a multi-turn agentic setting rather than only on single-turn outputs or final submissions.
  • The framework evaluates creativity through P-Creativity, H-Creativity, and usefulness on multi-turn ML engineering tasks.P-Creativity is novelty relative to an agent’s prior solutions, while H-Creativity is novelty relative to human solutions.
  • 10 ML competitions from MLE-Bench provide quantifiable usefulness metrics, human baselines, and solution spaces where novelty can be measured.The competitions include 877 to 3,747 public notebooks per competition for historical comparison.
  • LLM-as-a-Judge strongly correlates with human P-Creativity judgments, enabling automated evaluation at scale.The metric is applied across agent trajectories to assess how creativity changes over iterations.
  • Agents improve performance while P-Creativity declines over iterations, and higher P-Creativity has no reliable association with performance.LLM agents exhibit greater H-Creativity than medal-winning humans but fail to translate that novelty into performance gains.
  • The validated P-Creativity metric can serve as an optimization signal through novelty bonuses or adaptive exploration schedules.The paper frames novelty without usefulness as insufficient for autonomous discovery.

2 Related Work

Prior work evaluates LLM creativity through psychological tests, automated metrics, and domain-specific or iterative benchmarks. The paper positions its framework as addressing missing trajectory-level evaluation that separates psychological and historical novelty against broad human reference data.

  • Existing LLM creativity research spans psychological assessments, single-turn automated metrics, and domain-specific benchmarks.Examples include CreativeEval, NEOCODER, and CreativityPrism.
  • Concurrent benchmarks often measure performance or novelty without combining agent trajectories, human corpora, and separate psychological and historical novelty.InnoGym uses only 1–7 human solutions per task, InnovatorBench reports performance without a creativity metric, and CreativeBench scores one generation against one reference.
  • The framework addresses these gaps by separating P-Creativity from H-Creativity, validating metrics against human annotations, and tracking creativity across trajectories.Its human reference sets contain 877 to 3,747 solutions per task.

3 Methodology

The methodology defines creativity as novelty plus usefulness, separates psychological and historical novelty, and evaluates automated novelty metrics against human annotations. It then operationalizes trajectory episodes, feasibility, impact, and corpus-scale H-Creativity comparisons.

  • Unit of Analysis: Episodes begin with a new plan and end when code executes successfully with a valid submission score, while intermediate failures remain within the same episode.Each episode includes final executed code and the agent’s natural-language reasoning.
  • Creativity Framework: Creativity requires both originality and usefulness, so novel but impractical ideas and useful but conventional solutions are not creative.For research agents, usefulness includes demonstrably improved task performance.
  • Creativity Framework: P-Creativity measures novelty relative to an agent’s prior experience, whereas H-Creativity measures novelty relative to documented human solutions.In this setting, the human corpus consists of approaches explored by the Kaggle community.
  • Creativity Framework: H-Creativity is a special case of P-Creativity because community-novel ideas must first be novel to the agent.P-Creativity captures exploration beyond prior experience; H-Creativity requires reaching undocumented territory.
  • Usefulness: Usefulness is decomposed into feasibility and impact, with impact measuring improvement in the target metric after successful implementation.Only successfully executing episodes with valid scores enter the analysis.
  • Measuring Novelty: The study compares three automated metric families against human P-Creativity annotations before adopting validated metrics for analysis.The families are LLM-as-a-Judge, semantic distance, and conceptual novelty.
  • Measuring Novelty: H-Creativity uses a two-stage pipeline because directly judging every human solution is infeasible and semantic distance alone may confuse surface difference with algorithmic novelty.Semantic distance first retrieves candidate human solutions for further comparison.

4 Experimental Setup

The experiments evaluate two agent frameworks on 10 Kaggle-derived MLE-Bench competitions and compare agents with trajectory-based and corpus-based human references. Shared summarization and embedding procedures support fair agent–human comparisons.

  • Tasks: The benchmark contains 10 MLE-Bench competitions spanning diverse modalities and difficulty levels.Kaggle origins provide public human solution trajectories for direct comparison with agents.
  • Agent Frameworks: AIDE uses greedy tree search, while AIRA-Dojo represents agents as search policies over code artifacts with multiple search strategies.The two frameworks provide contrasting approaches to searching solution space.
  • Human Comparison Data: Episode-wise human trajectories include participants with at least 8 submissions spanning at least 60% of a competition timeline across 7 competitions.These trajectories support comparisons of P-Creativity and performance across episodes.
  • Human Comparison Data: The H-Creativity reference corpus contains all public notebooks with valid scores, totaling 877–3,747 notebooks per competition.This corpus represents the breadth of human approaches explored on each task.
  • Human Comparison Data: Medal-winning post-competition notebooks benchmark human H-Creativity against agent H-Creativity.These solutions were developed with full access to leaderboard results and community knowledge.
  • Implementation: Human and agent code pass through the same LLM summarization and embedding pipeline for fair comparison.

5 Results

Automated evaluation reliably tracks P-creativity, while agent trajectories show improving performance alongside declining novelty. Agents often explore beyond human solutions, but refinement and search strategies do not reliably convert novelty into impact.

  • Automated P-creativity measurement: r = 0.732 was the strongest correlation between automated metrics and human P-creativity annotations, achieved by LLM-as-a-Judge with GPT-5.The judge outperformed other judge models and embedding-based approaches.
  • Automated P-creativity measurement: Correlation remained strong after independently removing prompt components, indicating that the GPT-5 judge was not brittle to prompt design.Additional bias tests found no systematic bias across verbosity, position index, or self-preference.
  • Creativity and performance across iterations: All agents improved impact across episodes, with AIDE using GPT-5 showing the most consistent growth.AIRA-MCTS with Qwen started higher but plateaued, while the overall upward trend held across configurations.
  • Creativity and performance across iterations: P-creativity declined universally as episodes progressed, revealing a shift from exploration to exploitation.AIRA-MCTS operated at persistently low P-creativity, and all AIRA-Dojo strategies fell below a score of 1 within the first few episodes.
  • Creativity and performance across iterations: Strategic exploration fell from ∼75% to ∼25% of behavior instances, while parametric refinement rose from ∼15% to over 50%.Most strategic thinking occurred in the first quarter of runs, after which agents mainly refined their initial approach.
  • Creativity and performance across iterations: Different AIRA-Dojo search strategies converged to similar P-creativity and impact levels within a few episodes.The convergence suggests that surrounding agent scaffolding, including solution mutation, context transfer, and effective action space, dominates long-run behavior.
  • Novelty relative to human solutions: GPT-5 reached H-creativity 1.423 versus 0.744 for gold-medal humans, yet only 21.25% of GPT-5 runs achieved medal-level performance.Both Qwen3-32B configurations were statistically indistinguishable from gold-medal humans in H-creativity.
  • Novelty relative to human solutions: 87.0% of high-novelty, low-impact episodes led to refinement, but 53.8% of refined episodes subsequently decreased H-creativity and median ∆Impact remained 0.000.Neither refinement nor abandoning the approach resolved the novelty-impact gap.

6 Conclusion

The paper finds that LLM agents can generate novel solutions, but novelty does not reliably improve performance. Autonomous discovery therefore requires balancing exploration with usefulness.

  • LLM-as-a-Judge correlates most strongly with human P-creativity judgments (r = 0.732), outperforming embedding-based and concept-based metrics.
  • P-creativity declines universally as agents shift from exploration to exploitation, without systematic gains in impact.
  • GPT-5 explores solution regions more novel than gold-medal humans, yet only 21.25% of its runs reach medal-level performance.
  • Genuine autonomous discovery requires solutions that are both novel and useful; neither pure exploitation nor pure exploration alone produces breakthrough discovery.

A Future Directions and Limitations

The paper identifies P-creativity as an actionable optimization target while outlining limits to scaling evaluation, open-ended applicability, corpus completeness, and training-data isolation. Its human annotations show reliable creativity judgments despite subjectivity.

  • Future Directions: P-creativity declines structurally as agents converge toward refinement, motivating novelty bonuses, adaptive exploration schedules, and joint usefulness-diversity objectives.The proposed interventions penalize similarity to prior episodes and increase diversity when P-creativity falls below a threshold.
  • Limitations: Trajectories longer than 9 episodes make full-history evaluation computationally infeasible, motivating episode summarization or compressed prior-state representations.The current evaluation keeps full histories within most LLM context windows only for trajectories of up to 9 episodes.
  • Limitations: Open-ended research evaluation remains unresolved because such settings lack both quantitative usefulness metrics and bounded human solution corpora.The paper identifies surrogate usefulness signals and richer reference corpora as requirements for extension.
  • Limitations: Public Kaggle origins create possible training-data leakage, although H-creativity patterns provide evidence against pure strategy memorization and cannot fully verify isolation.Future validation on competitions released after model training cutoffs could further test this concern.
  • Limitations: H-creativity measures novelty relative to attempted Kaggle solutions rather than the full ML literature, using the public notebook corpus as a tractable proxy.The proxy is intended to represent approaches humans actually attempted on the specific task.
  • Evaluation Reliability: Three trained annotators evaluated 300 episodes using a 0–4 Boden-based ordinal rubric, with majority-vote labels and Krippendorff’s α (ordinal) = 0.724.The rubric ranged from routine repetition to transformational reframing, and annotators judged novelty against prior episodes in the same run.

B.2 Detailed Metric Formulations

The paper evaluates episodic novelty with LLM judgment, semantic-distance metrics, and concept-based metrics, while documenting limitations of alternative measures and episode extraction.

  • B.2 Detailed Metric Formulations: Seven automated metrics were evaluated to capture episodic novelty.The metric suite includes LLM-as-a-Judge, semantic-distance measures, and concept-level measures.
  • B.2.1 LLM-as-a-Judge: LLM-as-a-Judge scores each episode from 0 to 4 by comparing its plan and code with all previous episodes.The prompt includes a creativity rubric, the current episode, prior episodes, and structured output fields.
  • B.2.1 LLM-as-a-Judge: The judge’s input combines executed code, the agent’s summary, and an LLM-generated episode summary, with early code dropped after context exceeds 128,000 tokens.Plans and summaries are retained so algorithmic approaches remain available to the judge.
  • B.2.2 Semantic Distance: Semantic-distance metrics encode plans and executed code, then define distance-based novelty as 1 − sim(v_i, v_j).The paper compares nearest-neighbor, centroid, mean, and graph-edit-cost formulations.
  • B.2.2 Semantic Distance: Graph-edit cost inserts an episode into a similarity graph, but its higher cost can reflect many creative combinations rather than lower novelty.Edges connect sufficiently similar episodes, using threshold τ = 0.4.
  • B.2.3 Conceptual Metrics: Concept metrics extract high-level algorithmic concepts and measure novelty through fuzzy membership or set difference.Fuzzy membership averages semantic novelty across concepts, whereas set membership rewards concepts absent from all prior episodes.
  • B.2.3 Conceptual Metrics: Concept-level measures depend strongly on extraction quality because no standard concept vocabulary exists.The fuzzy metric uses σ = 0.3 by default or estimates bandwidth from prior nearest-neighbor distances.
  • B.2.4 Other metrics considered but not measured: Semantic entropy was rejected as infeasible, while surprisal was rejected because memory loss can make revisiting an old approach appear novel.Episode extraction records planning, code, debugging, execution results, scores, timestamps, and summaries; failed attempts remain within an episode.

C Task-Selection and Competition Details

The study selects 10 diverse MLE-Bench competitions to combine measurable task performance with rich, bounded human solution corpora for creativity evaluation.

  • Selection criteria: Ten MLE-Bench competitions were selected using human-data richness, computational feasibility, and task diversity criteria.The criteria target active public participation, datasets of approximately 10GB or less, and multiple modalities and difficulty levels.
  • Selected competitions: The selected competitions span four modalities and Low to Medium difficulty, with 877 to 3,747 notebooks and datasets up to 10.22GB.All 10 competitions support H-creativity evaluation, while highlighted rows support episode-wise trajectory analysis.
  • Why Kaggle competitions?: Kaggle competitions provide scalar performance metrics and bounded human corpora for measuring novelty against approaches actually attempted on each task.The public notebook corpus is used as a tractable proxy for the human solution space.
  • Human data: Human solution trajectories were constructed from Meta-Kaggle and Meta-Kaggle-Code datasets.These datasets provide public notebooks and submission histories for comparing agent and human creative processes.

D.1 Episode-wise Trajectories for RQ2

The trajectory analysis constructs comparable agent and human episodes, uses temporal and corpus-based references for H-creativity, and combines retrieval with LLM judgment for scalable novelty assessment.

  • Human trajectories: Seven competitions had multiple human participants with sufficient submission histories for trajectory analysis.Eligible humans required sustained engagement comparable to agents producing 8–10 episodes over 8 hours.
  • Human trajectories: Eligibility required at least 8 valid scored submissions spanning at least 60% of the competition timeline.Multiple same-day submissions were reduced to the highest-scoring submission, yielding 5–12 eligible humans per competition.
  • Scope caveat: The RQ2 human corpus captures strong competitors but not necessarily top-1 competitors because many top performers keep notebooks private until deadlines pass.This caveat limits the observed human creative processes used for trajectory comparison.
  • H-creativity corpora: The broader public-notebook corpus provides the reference set for evaluating whether agents explore regions humans did not document.All public notebooks with valid scores are included for H-creativity, spanning 877 to 3,747 notebooks per competition.
  • Medal-winning baseline: Medal-winning post-competition notebooks provide a strong human baseline because their authors had access to competition data, leaderboard results, and shared knowledge.The corpus includes publicly available bronze-, silver-, and gold-level notebooks.
  • Standardized representations: The agent and human representations standardize code, approach summaries, scores, and timestamps for direct comparison.Agent episodes include comparable plans, executed code, performance scores, and timing information.
  • Two-stage H-creativity pipeline: The H-creativity pipeline retrieves nearest human neighbors using embeddings, then has GPT-5 judge substantive divergence across six methodological dimensions.Retrieval uses shared LLM summaries and Qwen3-Embedding-4B embeddings; the judge scores novelty from 0–4.
  • Temporal reference set: Human H-creativity uses a temporal reference set containing only notebooks submitted before each notebook’s submission date.This avoids comparing an earlier human solution with approaches that appeared later.

F Cognitive Behavior Analysis

The cognitive-behavior analysis links declining P-creativity with a shift from strategic exploration toward parameter tuning and bug fixing during agent runs.

  • Behavior categories: Reasoning traces are classified into strategic exploration, parametric refinement, and error/bug fixing.DeepSeek-V3.2 classifies instances in AIDE(Qwen3-32B) runs.
  • Run progression: Strategic exploration falls from around 75% to roughly 25% of behavior instances by run end.This measures the distribution of classified behavior across run progression.
  • Run progression: Parametric refinement rises from 15% to over 50%, while error/bug fixing grows from 5% to 25%.The changes accompany the decline in strategic exploration.
  • Interpretation: Agents increasingly tune parameters and fix bugs after establishing a working solution instead of exploring alternative approaches.Most strategic thinking occurs in the first quarter of runs.
  • Novelty–performance disconnect: Ninety-two episodes combine top-20th-percentile H-creativity with bottom-20th-percentile impact for analysis of the novelty–performance disconnect.The analysis targets episodes that are highly novel relative to human solutions but weak in impact.

G.1 Agent Behavior After Reaching Novel Regions

After reaching highly novel solutions, agents usually refine rather than abandon them, but neither path reliably converts novelty into impact. Refinement often reduces novelty without measurable impact gains, while low-impact episodes are not selectively abandoned.

  • Trajectory choices: 87.0% of episodes refined high-H-creativity approaches, while 13.0% abandoned them.None moved toward the nearest human solution; 100% maintained or increased distance from the human baseline.
  • Refinement outcomes: 53.8% of REFINE episodes decreased H-creativity, with median ∆Impact = 0.000 across 80 episodes.Most refinements were minor or moderate: 51.1% MINOR and 35.9% MODERATE.
  • Refinement outcomes: ABANDON episodes had a 75.0% novelty-decrease rate and median ∆Impact of 0.083 across 12 episodes.Switching approaches therefore offered only marginal improvement relative to the novelty-impact gap.
  • Selective behavior: Low-impact episodes were abandoned less often than high-impact episodes, at rates of 6.5% and 16.4%, respectively.This pattern held across all three agent configurations and indicates weak selective discrimination between promising and unpromising novel regions.

I Per-Task Breakdown

Per-task analyses broadly reproduce the aggregate creativity patterns: P-creativity generally declines across episodes, while agents often exceed gold-medal human solutions in H-creativity. GPT-5 shows the broadest per-task H-creativity advantage, though one task falls below humans for every agent.

  • P-creativity: Declining P-creativity is the dominant trend across agent configurations and most individual tasks.The per-task breakdown uses linear-fit slopes over episodes 1–9.
  • H-creativity: GPT-5 exceeds the gold-medal human H-creativity baseline on 9 of 10 tasks.AIRA (Qwen3-32B) exceeds it on 7 of 10 tasks, while AIDE (Qwen3-32B) exceeds it on 5 of 10.
  • H-creativity: Ventilator pressure prediction is the only task where all agents fall below the human H-creativity baseline.The figure-level breakdown also reports GPT-5 above the human baseline on most tasks, with Qwen3-32B configurations closer to it.
Loading 2608.30047v1…