PaperlayerView exact revision on arXiv

Source-linked AI summary

InnoGym: Benchmarking the Innovation Potential of AI Agents

Jintian Zhang, Kewei Xu, Jingsheng Zheng, Zhuoyun Yu, Yuqi Zhu, Yujie Luo, Lanning Wei, Shuofei Qiao, Lun Du, Da Zheng, Shumin Deng, Huajun Chen, Ningyu Zhang

arXiv:2512.01822v2cs.CLcs.AIcs.CVcs.LGcs.MA

TL;DR

Existing benchmarks largely assess answer correctness, leaving the methods behind solutions overlooked. InnoGym combines performance gain and novelty in a benchmark of standardized tasks with a unified execution environment. Experiments find that novelty without robustness does not translate into meaningful performance gains.

  • Problem

    Most existing benchmarks assess answer correctness while overlooking the methods used to reach solutions.

    Source: §1 · p. 1

  • Method

    InnoGym evaluates performance gain against best-known solutions and methodological novelty across 18 curated tasks, with iGym providing a unified execution environment.

    Source: §1 · p. 1

  • Results

    Some agents show high novelty, but their lack of robustness prevents those innovations from yielding meaningful performance gains.

    Source: §1 · p. 2

  • Takeaways & Limitations

    InnoGym provides a reproducible, cross-domain platform for evaluating both creativity and reliable performance.

    Source: §6 · p. 10

    The benchmark focuses on Improvable Tasks with clear evaluation pipelines, excluding solved problems and open-ended exploratory problems.

    Source: B LIMITATIONS · p. 16

Abstract

from arXiv · show

LLMs and Agents have achieved impressive progress in code generation, mathematical reasoning, and scientific discovery. However, existing benchmarks primarily measure correctness, overlooking the diversity of methods behind solutions. True innovation depends not only on producing correct answers but also on the originality of the approach. We present InnoGym, the first benchmark and framework designed to systematically evaluate the innovation potential of AI agents. InnoGym introduces two complementary metrics: performance gain, which measures improvement over the best-known solutions, and novelty, which captures methodological differences from prior approaches. The benchmark includes 18 carefully curated tasks from real-world engineering and scientific domains, each standardized through resource filtering, evaluator validation, and solution collection. In addition, we provide iGym, a unified execution environment for reproducible and long-horizon evaluations. Extensive experiments show that while some agents produce novel approaches, their lack of robustness limits performance gains. These results highlight a key gap between creativity and effectiveness, underscoring the need for benchmarks that evaluate both.

1 INTRODUCTION

Correctness-focused benchmarks overlook how agents reach solutions, despite innovation depending on both results and methods. InnoGym evaluates performance gain and novelty across standardized real-world tasks, and experiments find that novelty without robustness does not yield meaningful performance gains.

1 INTRODUCTION

  • Existing benchmarks primarily judge whether answers are correct, overlooking methodological differences between solutions.Agents can reach the same correct answer using entirely different approaches.Source: §1 · p. 1
  • InnoGym measures innovation through performance gain over the best-known baseline and novelty as methodological difference from prior solutions.Its framework formalizes each task using a problem instance, solution space, performance measure, and dissimilarity measure.Source: §1 · p. 1
  • iBench includes 18 Improvable Tasks from real-world engineering and scientific problems with room to improve performance and methodology.The tasks are distinguished from solved problems and exploratory problems lacking human baselines or reliable validation.Source: §1 · pp. 1–2
  • Each task is standardized through resource checks, evaluator validation, solution collection, validator construction, and dataset partitioning.These stages are intended to support fairness and reproducibility.Source: §1 · p. 2
  • iGym provides a unified agent execution environment for robust tool use and long-horizon problem solving across diverse systems.The environment supports consistent comparisons.Source: §1 · p. 2
  • Some agents demonstrate high novelty, but their lack of robustness prevents these innovations from translating into meaningful performance gains.The experiments identify a gap between originality and effectiveness in real-world scientific and engineering problems.Source: §1 · p. 2

2 DEFINING AND MEASURING INNOVATION

The section defines innovation as feasible solutions that create value through better performance or distinct methods, and organizes its evaluation around performance gain and novelty. It formalizes tasks and distinguishes innovation regimes and task types by their known solutions and improvement potential.

2.1 TASK AND NOTATION

  • A task is formalized as T = (P, S, V, D), with performance combining feasibility and objective score, and distance measuring solution dissimilarity.Evaluation uses the changing set of known solutions as its empirical reference when the optimum is unknown, intractable, or nonexistent.Source: §2.1 · p. 3

2.2 DEFINITION AND EVALUATION OF INNOVATION

  • Innovation requires feasibility and meaningful differentiation from known solutions, measured through performance gain and novelty.Performance gain compares against the known-solution frontier; novelty measures dissimilarity to prior solutions and is awarded only to feasible ones.Source: §2.2 · p. 3
  • High performance gain with high novelty is breakthrough innovation; high gain with low novelty is performance innovation, while near-zero gain with high novelty is conceptual innovation.Highly novel solutions that perform substantially worse, as well as solutions that are neither better nor different, count as unsuccessful exploration.Source: §2.2 · p. 4

2.3 DISCUSSION: A TAXONOMY OF INNOVATIVE TASKS

  • The taxonomy distinguishes solved problems, improvable problems, and exploratory problems by their known solutions and improvement potential.Solved problems reward novelty at a known optimum; improvable problems allow performance gains or different methods at comparable performance; exploratory problems have no known feasible solutions, so the first feasible solution is a breakthrough.Source: §2.3 · p. 4

3 INNOGYM: BENCHMARK AND SYSTEM (IBENCH & IGYM)

InnoGym combines iBench, a benchmark of 18 standardized improvable tasks, with iGym, an execution environment for diverse agent systems and long-horizon problem solving. Its construction filters and standardizes tasks, formalizes what agents can access, and evaluates performance and novelty in sequence.

3 INNOGYM: BENCHMARK AND SYSTEM (IBENCH & IGYM)

  • iBench contains 18 improvable tasks from real-world engineering and theoretical problems, excluding solved tasks and exploratory tasks without human baselines or reliable validation.The included tasks retain room for improvement in both solution quality and methodology.Source: §3 · p. 4

3.1 TASK SOURCES AND TWO-STAGE FILTERING

  • 197 collected tasks are filtered to 18 balanced improvable tasks through resource and evaluator checks, with scoring standardized for executability, correctness, and absolute metrics.The initial pool draws on public competitions and classic science and engineering problems; unfixable evaluators are removed and domains are balanced.Source: §3.1 · pp. 4–5, Figure 2 · p. 5

3.2 ENHANCEMENT AND STANDARDIZATION

  • Task standardization covers specifications and environments, validators, collected solutions, normalized evaluators, and visible-versus-hidden data partitions.Evaluator normalization addresses absolute scoring, executability, and correctness; tasks that fail consistency checks are discarded.Source: §3.2 · p. 5

3.3 TASK FORMALIZATION

  • Agents receive visible task information and validators, while evaluation data, performance evaluators, and known solutions remain hidden.The task is formalized as a problem, solution space, performance measure, and distance function for novelty.Source: §3.3 · pp. 5–6

3.4 EVALUATION PIPELINE

  • The evaluation pipeline collects an agent’s solution artifact, scores it only if feasible, then compares extracted features with known solutions to produce novelty.Performance gain and novelty are complementary measures, and both are required for a task to count as innovative success.Source: §3.4 · p. 6

3.5 IGYM: A UNIFIED AGENT EXECUTION ENVIRONMENT

  • iGym provides a common abstraction for diverse agent systems, supporting workflow-style and agent-style interaction with environments, tools, and resources.It is designed for long-running tasks and includes robust recovery, native concurrency, and consistent tool management.Source: §3.5 · p. 6

4 EXPERIMENTS

The experiments compare agent frameworks under a shared evaluation setup and find substantial gaps from human state of the art, with robustness limiting the value of novelty. Controlled analyses examine how prior solutions, time, backbone models, and sampling temperature relate to performance gain and novelty.

4.1 EXPERIMENTAL SETUP

  • The main evaluation uses three frameworks in iGym, measures performance gain and judge-rated methodological novelty, and evaluates 10 of 18 tasks with up to 12 hours per configuration.Each configuration runs three times, and the best valid score is reported; Table 2 compares the frameworks with leaderboard bounds.Source: §4.1 · pp. 7–8, Table 2 · p. 8

4.2 MAIN RESULTS

  • No evaluated agent surpassed human state-of-the-art solutions, and all agents failed to produce valid executable solutions on CDML and PTTALC.Table 2 compares the three frameworks with leaderboard bounds on the 10 main iBench tasks.Source: Table 2 · p. 8, §4.2 · p. 8
  • MLab leads in both Performance Gain and Novelty, while CodeAct and AIDE lag on both; CodeAct nears state of the art on CirclePacking but that strength does not generalize.The authors attribute the weaker profiles likely to handling complex file structures and tool use.Source: §4.2 · p. 8
  • Comparable innovation levels across frameworks coexist with divergent performance; mid-to-high novelty can accompany low scores, making reliable execution the foremost challenge.RCIC and TrojanDetection illustrate this mismatch; the authors identify correctness and robustness as bottlenecks.Source: §4.2 · p. 8

4.3 EXPERIMENTAL ANALYSIS

  • Starting from a Gemini-2.5-Pro solution with sum radii 2.59 and ratio 0.98, AIDE finds better outcomes; novelty first rises, then falls as performance improves toward a local optimum.Figure 5 maps performance gain to complex-plane magnitude and normalized novelty to angle, revealing directional shifts that scalar novelty can obscure.Source: §4.3 · p. 9, Figure 5 · p. 9
  • Over extended search, performance gain tends to improve while novelty decreases; gain remains non-negative, which the authors interpret as a stable, monotonically improving refinement process.They relate the pattern to diminishing returns and methodological convergence as solutions improve.Source: §4.3 · pp. 9–10
  • Gemini-2.5-Pro scores 2.49 and hypothetical GPT-5 scores 2.44, approaching AlphaEvolve's 2.65; DeepSeek-v3.1 scores 2.40.The authors conclude that agent frameworks amplify rather than substitute for the base model's reasoning and coding abilities.Source: Figure 6 · p. 9, §4.3 · p. 10
  • Low sampling temperatures yield the highest Performance Gain, while novelty rises with temperature; the reported sweet spot is 0.5–0.75 for near-optimal performance and higher novelty.Figure 6 covers execution-time budgets, backbone models, and sampling temperatures; the cited text reports three runs for the first two analyses.Source: Figure 6 · p. 9, §4.3 · p. 10

5 RELATED WORK

Prior evaluations span correctness-focused capabilities, performance-driven improvable tasks, idea generation, and agents that solve concrete problems. The paper identifies a gap in distinguishing methodological novelty from solution value.

Evaluation for ML Engineering and Scientific Discovery

  • Early LLM benchmarks emphasized mathematical reasoning and code generation, typically judging correctness through unit tests or exact-match answers.The cited examples include Hendrycks et al. (2021), Cobbe et al. (2021), Chen et al. (2021), and Jain et al. (2025).Source: §5 · p. 10
  • Performance-driven benchmarks such as MLE-Bench evaluate solution value, but may conflate that value with methodological novelty.Similar performance-focused evaluation is described for ML engineering, scientific discovery, and data science.Source: §5 · p. 10

LLM Agents of Innovation

  • Idea-generation benchmarks and systems assess research ideation, while their ideas' downstream value often remains speculative; problem-solving agents instead execute ideas on concrete problems.AlphaEvolve (Novikov et al., 2025) is cited for superior solutions to matrix multiplication and sphere packing.Source: §5 · p. 10

6 CONCLUSION

The conclusion presents InnoGym as a benchmark and framework that evaluates both performance gain and methodological novelty. Its experiments indicate that novelty may occur without robustness.

6 CONCLUSION

  • InnoGym combines performance gain and methodological novelty to evaluate both effectiveness and creativity beyond correctness-only assessment.The conclusion describes it as a benchmark and framework for AI-agent innovation potential.Source: §6 · p. 10
  • The benchmark contains 18 standardized tasks and a unified execution environment for reproducible cross-domain comparisons.Source: §6 · p. 10
  • Experiments find that current agents often achieve novelty without robustness, leaving a gap between creativity and reliable performance.Source: §6 · p. 10

ETHICS STATEMENT

The statement reports that the study used publicly available data and affirms adherence to ethical standards and research best practices.

ETHICS STATEMENT

  • All study data were derived and synthesized from publicly available sources, with no proprietary or confidential information used.Source: ETHICS STATEMENT · p. 11
  • The authors state that data-source references are accurately and appropriately cited throughout the paper.Source: ETHICS STATEMENT · p. 11
  • The authors encourage fair, transparent, responsible dataset use and prohibit uses that could cause harm or negatively affect society.Source: ETHICS STATEMENT · p. 11

B LIMITATIONS

InnoGym’s limitations concern its task scope and innovation metrics, while iGym’s design targets long-horizon execution and task diversity is summarized by reference-solution distances.

B LIMITATIONS

  • The benchmark covers only Improvable Tasks with clear evaluation pipelines, excluding solved and open-ended exploratory problems.Its performance-gain and novelty metrics may omit dimensions such as efficiency, interpretability, and long-term impact; novelty also relies on a finite set of known solutions.Source: B LIMITATIONS · p. 16

C IGYM

  • iGym’s unified SDK addresses needs for diverse agent systems, long-horizon recovery, resource management, and concurrency.Its environment supports parallel long-running tasks, progress monitoring, and real-time results; workflow runs replay recorded calls, while agent runs resume from saved state.Source: C IGYM · p. 17

D TASK

  • Div(T) summarizes the average pairwise method-level distance among a task’s known solutions, with larger scores indicating a wider spread in method space.The score lies in [0, 100]; it is undefined when only one reference solution is available.Source: D TASK · p. 17

D.1 DETAIL TAXONOMY OF INNOVATIVE TASKS

The taxonomy classifies each concrete task instance by feasibility and remaining headroom under its specified goal; the setup also evaluates several complementary agent scaffolds.

D.1 DETAIL TAXONOMY OF INNOVATIVE TASKS

  • Table 3 reports each task’s source benchmark, number of known reference solutions, and diversity score based on pairwise method-level distance.The diversity score describes how widely the reference solutions occupy method space.Source: Table 3 · p. 18
  • A task is solved when a known feasible solution is optimal, improvable when a feasible known solution leaves objective headroom, and exploratory when no known solution is feasible.These categories are defined relative to the task instance and its human-defined goal.Source: §D.1 · pp. 17–18
  • The categories depend on the chosen validator and value function, so changing the explicit objective can reclassify the same real-world problem.Under SWE-Bench’s test-suite objective, a passing human patch makes a typical instance solved; requiring minimal edits or broader robustness could make it improvable.Source: §D.1 · p. 18

E.1 SETUP

  • MLAB (Huang et al., 2024), CODEACT (Wang et al., 2025), and AIDE (Jiang et al., 2025) represent workflow planning, code-centric actions, and search-based exploration, respectively.Together, these scaffolds provide complementary design choices for evaluating innovation on InnoGym.Source: §E.1 · p. 19

E.2 STATISTICAL ANALYSIS FOR TABLE 2

The analysis imputes failures pessimistically and compares agent frameworks across tasks using macro-averages and paired bootstrap tests. MLAB leads on normalized ratio, while all frameworks remain far from best-known solutions on average.

E.2 STATISTICAL ANALYSIS FOR TABLE 2

  • A task with no valid submission receives R = −1 and N = 0, ensuring failures are included in framework scores.Under this encoding, every framework has scores on all 10 main tasks.Source: §E.2 · p. 19
  • The analysis computes task-level macro-averages and estimates uncertainty with 10,000 non-parametric bootstrap resamples over tasks.Paired framework comparisons bootstrap task-wise metric differences and report 95% confidence intervals and approximate two-sided p-values.Source: §E.2 · pp. 19–20
  • Even under pessimistic treatment of failure cases, all three agent frameworks remain far from best-known solutions on average (ratios close to −1), with MLAB attaining the best normalized ratio and highest average novelty.Source: §E.2 · p. 19
  • MLAB’s macro-average ratio R is significantly higher than CODEACT’s and AIDE’s, while its highest mean novelty N is not significantly different at the 0.05 level.The authors use R as the primary metric for formal framework comparisons and treat N differences as descriptive trends.Source: Table 5 · p. 20, §E.2 · p. 20

E.3 OTHER ANALYSIS

The analyses report persistent submission failures and a novelty–performance trade-off, then describe an agent-based distance measure and its validation against human judgments.

E.3 OTHER ANALYSIS

  • 2 of the 10 sampled tasks, CDML and PTTALC, had no valid submissions from any agent.Table 6 reports submission success rates across three runs; the authors characterize these challenges as requiring runtimes spanning tens of hours.Source: Table 6 · p. 20, §E.3 · p. 20
  • On three tasks, prompting AIDE to prioritize innovation increased novelty but consistently decreased Performance Gain.The novelty increase was significant for CirclePacking; Table 7 compares standard and innovation prompts.Source: §E.3 · p. 20, Table 7 · p. 21

F VALIDATING THE DISTANCE FUNCTION D AND NOVELTY METRIC N

  • DAGENT instantiates the solution-distance function with an agent-based pipeline that extracts standardized solution representations and compares method-level differences.The authors validate it with human judgments and code-equivalence and cross-subfield triplet experiments.Source: F VALIDATING THE DISTANCE FUNCTION D… · p. 21

F.1 RECAP: NOVELTY EVALUATION PIPELINE

  • DAGENT normalizes each solution into summary.md and pseudocode.tex, then scores differences across fixed method dimensions on a 0–4 scale and aggregates them to 0–100.The representations omit incidental details such as file layout and naming conventions; the main paper uses DAGENT for novelty scores.Source: §F.1 · p. 21

F.2 TRIPLET-BASED VALIDATION PROTOCOL

  • The validation triplets pair a base solution A with alternatives B and C chosen to be relatively closer and farther, then compare agent and human distance scores and judgments.Agreement is assessed through score correlations and whether agent and human judgments agree on which alternative is more novel or roughly tie; experiments use EquiBench and human-constructed triplets from three AI subfields.Source: §F.2 · p. 22

F.3 EXPERIMENT 1: EQUIBENCH CODE-LEVEL SANITY CHECK

Across code-level and cross-domain tests, DAGENT distinguished meaningful methodological differences from cosmetic edits and tracked human judgments of novelty. Its results support using it to compare novelty at both the implementation and high-level method-design levels.

F.3 EXPERIMENT 1: EQUIBENCH CODE-LEVEL SANITY CHECK

  • Across 50 triplets, DAGENT scored OJ A algorithmic variants 9.75 versus 1.00 for OJ V cosmetic variants, and matched human novelty preferences on 6/8 annotated triplets.The human evaluation covered eight triplets; agent and human mean scores were around 16 for (A, C) and essentially 0 for (A, B).Source: §F.3 · p. 23

F.4 EXPERIMENT 2: HUMAN-COLLECTED METHOD TRIPLETS

  • Across three domain triplets, experts and DAGENT consistently rated cross-paradigm C as more novel than within-paradigm B, with 3/3 triplet-level agreement despite the tiny sample.Their average scores were close in magnitude, and score-level correlations were described as essentially perfect.Source: §F.4 · p. 23

F.5 OVERALL SUMMARY

  • Across code-level and cross-domain tests, DAGENT distinguished method-level changes from cosmetic edits and tracked human intuitions about novelty at both levels.The overall summary combines the EquiBench sanity check with cross-domain method triplets.Source: §F.5 · pp. 23–24

G DETAILS OF BENCHMARK CONSTRUCTION

This section describes how benchmark construction creates a validated, reproducible basis for comparing solutions and operationalizes novelty through structured, task- and time-relative references and similarity assessments across algorithmic, optimization, data-processing, and feature-engineering choices.

G.1 SOLUTION COLLECTION

  • For each task, the team gathered candidate methods from authoritative sources and public solutions, then independently collected and filtered them for reproducibility before abstracting retained solutions for novelty analysis.Classical-problem searches used textbooks and surveys, while competition searches included high-ranking public entries; the final set excluded solutions that could not run or reproduce reported performance.Source: §G.1 · pp. 24–25

G.2 EVALUATOR NORMALIZATION

  • Evaluator normalization addressed heterogeneous implementations through a unified Python interface, while correctness checks compared known-solution scores with reported results and tested whether high-performing solutions outscored trivial baselines.Non-Python evaluators were containerized and called through Python wrappers; evaluators failing checks were debugged until they passed.Source: §G.2 · p. 25, §G.2.2 · p. 26, §G.2.3 · p. 26

G.2.1 ABSOLUTENESS

  • Rank-based ROADEF scores were converted to absolute scores by fixing leaderboard extrema and applying logarithmic normalization, enabling direct comparison of new valid solutions with existing ones.The rationale is that rankings alone cannot measure the magnitude of performance improvement over the state of the art.Source: §G.2.1 · p. 25
  • Normalized absolute scores strongly preserved the original ROADEF leaderboard ordering, according to Spearman and Kendall rank-correlation validation.Table 13 presents the correlations; the supplied passages do not provide their numerical values.Source: §G.2.1 · p. 25, Table 13 · p. 26

G.3 VALIDATOR CONSTRUCTION

  • Task-specific validators reject invalid code or answer files by checking interfaces, execution and output types, formats, schemas, and field constraints; procedural implementation makes outcomes deterministic and reproducible.Code is tested on a small fixed input, while structured answer files are checked against required formats and value constraints.Source: §G.3 · p. 27

H.1 EXTRACTION PROMPT

  • The extraction prompt directs a model to reconstruct each solution’s high-level methodology from its repository and any accompanying paper, producing a structured summary and pseudocode.It focuses on problem context, modeling choices, data flow, training strategy, and evaluation while omitting low-level implementation details.Source: §H.1 · p. 27

H.2 COMPARISON PROMPT

  • The comparison prompt scores similarity between an agent solution and a baseline across six dimensions, with brief rationales supporting quantitative, reproducible comparisons.Each dimension receives a score from 0 (completely similar) to 4 (completely different).Source: §H.2 · p. 27

I.1 PROPERTIES OF INNOVATION

  • Innovation is defined relative to the task and known-solution set, and accepted innovative solutions enter the next baseline, reducing the measured novelty of similar future approaches.This update models innovation as movement from exploration to optimization and, eventually, a solved state.Source: §I.1 · pp. 27–28

5. Algorithm & Optimization

  • The algorithm-and-optimization comparison asks whether solutions use similar algorithms and optimization techniques, including learning-rate schedules, regularization, and hyperparameter tuning.Source: §5 · p. 30

6. Data Processing & Feature Engineering

  • The data-processing comparison considers similarity in data cleaning, scaling, transformation, and feature selection, extraction, or construction.Source: §6 · p. 30
  • Each dimension receives a 0–4 similarity score with a short justification, and the JSON output includes dimension-level scores and justifications plus a total score.The provided example assigns a total score of 9.Source: §6 · pp. 30–31
Loading 2512.01822v2…