PaperlayerView exact revision on arXiv

Source-linked AI summary

SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories

Ben Bogin, Kejuan Yang, Shashank Gupta, Kyle Richardson, Erin Bransom, Peter Clark, Ashish Sabharwal, Tushar Khot

arXiv:2409.07440v1cs.AIcs.CLcs.SE

TL;DR

Researchers lack a benchmark for evaluating whether LLM agents can both set up and execute experiments in low-profile, under-documented research repositories. SUPER provides Expert, Masked, and Auto problem sets and evaluates outcome correctness or execution progress using gold solutions, landmarks, or approximate script success. The strongest agent solves only 46.1% of Masked sub-problems and 16.3% of end-to-end Expert tasks; agents handle well-specified bugs better than repository exploration.

  • Problem

    Researchers lack a benchmark for evaluating whether LLM agents can both set up and execute experiments in low-profile, under-documented research repositories.

    Source: §1 · pp. 1–2

  • Method

    SUPER provides Expert, Masked, and Auto problem sets and evaluates outcome correctness or execution progress using gold solutions, landmarks, or approximate script success.

    Source: §1 · p. 2, §3.1.1 · p. 4, §3.1.2 · p. 5

  • Results

    The strongest agent solves only 46.1% of Masked sub-problems and 16.3% of end-to-end Expert tasks; agents handle well-specified bugs better than repository exploration.

    Source: §1 · p. 2

  • Takeaways & Limitations

    SUPER provides a resource for measuring progress toward autonomous research-repository execution while exposing repository reasoning and code editing as core challenges.

    Source: §5 · p. 10

    The benchmark supports only Python and focuses mostly on text-based repositories, so broader repository-domain diversity could be beneficial.

    Source: Limitations · p. 10

Abstract

from arXiv · show

Given that Large Language Models (LLMs) have made significant progress in writing code, can they now be used to autonomously reproduce results from research repositories? Such a capability would be a boon to the research community, helping researchers validate, understand, and extend prior work. To advance towards this goal, we introduce SUPER, the first benchmark designed to evaluate the capability of LLMs in setting up and executing tasks from research repositories. SUPERaims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges (e.g., configuring a trainer), and 602 automatically generated problems for larger-scale development. We introduce various evaluation measures to assess both task success and progress, utilizing gold solutions when available or approximations otherwise. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios. This illustrates the challenge of this task, and suggests that SUPER can serve as a valuable resource for the community to make and measure progress.

1 Introduction

SUPER asks whether LLM agents can autonomously set up and execute experiments in under-documented research repositories, a difficult process involving configuration, dependency resolution, code changes, and reporting results. The benchmark shows substantial performance gaps and reveals that repository exploration is harder than well-specified troubleshooting.

Motivation

  • SUPER targets the realistic challenge of setting up and executing experiments in low-profile research repositories, where documentation and support may be limited.Tasks can require installing environments, changing configurations, resolving dependencies, fixing bugs, identifying commands, and understanding repository code.Source: §1 · pp. 1–2

Benchmark

  • The benchmark contains 45 expert tasks, 152 focused sub-problems derived through code masking, and 604 automatically generated tasks for broader development.The sub-problems target challenges including dependency conflicts, data configuration, hyper-parameters, runtime exceptions, and script execution.Source: §1 · p. 2

Evaluation

  • SUPER evaluates gold-answer correctness and partial progress through landmark states, while Auto tasks use successful key-script execution as an approximate metric.Landmarks include outputs indicating completed training or successful data loading.Source: §1 · p. 2

Results

  • On SUPER, the strongest agent solved only 46.1% of Masked sub-problems and correctly completed just 16.3% of end-to-end Expert tasks, while open-source models substantially lagged on both.Proprietary and open-source LLMs were evaluated as underlying models of strong baseline agents with access to file-editing tools.Source: §1 · p. 2

Analysis

  • Trajectory analysis finds agents handle well-specified exceptions and bugs better than tasks requiring repository and file exploration to understand code structure.The results identify repository reasoning as a core challenge for autonomous experiment execution.Source: §1 · p. 2

2 Related Work

SUPER extends code-execution benchmarking to end-to-end experimentation in low-profile research repositories, combining repository understanding, setup, interactive execution, and outcome-based evaluation. Its construction pipeline derives focused Masked problems from expert-authored tasks.

Coding benchmarks

  • The benchmark addresses natural programming scenarios involving external tools, APIs, debugging, repository issue resolution, and repository-context coding.SUPER applies these capabilities specifically to research-task execution rather than isolated function synthesis or competitive programming.Source: §2 · pp. 2–3

Comparison with related benchmarks

  • SUPER differs from related benchmarks by requiring repository comprehension, multi-file editing, environment setup, interactive execution, and outcome-based evaluation in low-profile repositories.Existing benchmarks often use popular repositories or single-script optimization and may not assess whether resulting metrics are correct.Source: §2 · p. 3, Table 1 · p. 3

LLM agents

  • SUPER introduces a benchmark domain for LLM agents assisting researchers with end-to-end tasks across arbitrary repositories.The benchmark complements prior agent work in games, web navigation, data science, machine learning, discovery, and coding.Source: §2 · p. 3

SUPER construction

  • Figure 2’s construction pipeline creates the Expert set from manually written tasks and expert solutions, then extracts focused Masked problems from those solutions.The benchmark also includes an Auto set of automatically generated problems for agent development.Source: Figure 2 · p. 3

3 Benchmark Construction

SUPER is constructed as a three-set benchmark for evaluating agents on realistic research-repository setup and execution, with validated expert tasks, focused masked sub-problems, and scalable automatically generated tasks. Its environments and evaluation measures capture both exact outcomes and incremental progress while accounting for practical execution constraints.

3 Benchmark Construction

  • SUPER comprises Expert, Masked, and Auto sets serving validated evaluation, focused challenge analysis, and scalable agent development.The Expert set contains manually written problems, the Masked set derives sub-problems from expert solutions, and the Auto set is automatically generated.Source: §3 · p. 3, §3.3 · p. 6, §3.4 · p. 7

3.1 Expert Set

  • 45 expert tasks target paper reproduction or modified experiments and are paired with deterministic solutions, reported outputs, dependencies, and recorded implementation decisions.Tasks are manually written from research repositories and reviewed for executability, completeness, and repeatability.Source: §3.1.1 · pp. 4–5, §3.1 · p. 4

3.1.1 Construction

  • The benchmark keeps experiments lightweight by avoiding GPU dependence and limiting execution to roughly 10 minutes, while retaining challenges such as configuration and CPU adaptation.Tasks use small models and constrained data or training settings, but these restrictions can require additional repository-specific changes.Source: §3.1.1 · p. 4

3.1.2 Evaluation

  • Expert tasks use outcome accuracy for exact answers and landmark evaluation for partial progress toward completing multi-step executions.Landmarks identify successful intermediate outputs, but a perfect landmark score does not guarantee correct results because actions may use incorrect settings.Source: §3.1.2 · p. 5

3.2 Masked Coding sub-problems Extraction

  • Masked sub-problems remove solution cells for a targeted issue, pre-execute independent prefix cells, and require agents to complete a clearly defined goal.This design supports fine-grained evaluation of technical challenges and mirrors interactive assistants helping users after some code has already been written or run.Source: §3.2 · pp. 5–6

3.3.1 Construction

  • Auto tasks are generated from filtered repository READMEs, but a sample found only 81% feasible because missing resources and conceptual flaws remained.Filtering excludes repositories requiring GPUs, unavailable APIs, or lacking usable datasets and models.Source: §3.3.1 · pp. 6–7

3.3.2 Evaluation

  • Without expert solutions, Script-Executed heuristically checks whether the requested script executes without exceptions for a minimum duration, assessing setup and execution rather than outcomes or landmarks.The minimum duration helps ensure the script succeeded rather than failed silently, although this method does not guarantee perfect evaluation and was found surprisingly effective.Source: §3.3.2 · p. 7

3.4 SUPER Benchmark

  • The Masked set contains 152 sub-problems with gold outputs and landmarks, while the Expert set contains 45 tasks and the Auto set contains 604 problems from unique repositories.The benchmark also reports an average of 44.3 lines of code and 14.4 cells per expert solution, indicating long execution trajectories.Source: §3.4 · p. 7

4 Experiments

SUPER evaluates notebook-based agents across expert, masked, and automatically generated research-repository tasks, measuring both final answers and intermediate progress. Results show substantial difficulty, with performance varying by tools, model strength, task type, and repository-analysis demands.

4.1 Baselines

  • Agents were evaluated with ReAct, ReAct-SUPER, and SWE-Agent, using notebook execution tools and, for some agents, file-editing or file-browsing actions.ReAct-SUPER adds an edit action, while SWE-Agent also provides tools for reading and scrolling through files.Source: §4.1 · p. 8

4.2 Results

  • 16.3% of Expert tasks were solved by the strongest agent, while agents still achieved 46.1% accuracy on Masked sub-problems.Expert-set accuracies ranged from 12.2% to 16.3%, whereas the strongest Masked-set result reached 46.1%; landmark scores indicate additional partial progress.Source: §4.2 · p. 9, Table 6 · p. 9
  • SWE-Agent outperformed ReAct-SUPER with GPT-4o but performed slightly worse with weaker models, while Mixtral and Llama scored substantially lower on Expert and Masked sets.The results suggest weaker models were less effective at leveraging SWE-Agent’s tools.Source: §4.2 · p. 9
  • The Auto-set model and agent ranking was mostly consistent with the Masked set, supporting its use for future agent development.The Auto set uses a script-execution proxy because gold solutions are unavailable.Source: §4.2 · p. 9
  • Adding file editing improved ReAct-SUPER’s landmark score from 65.7% to 72.5% and accuracy from 37.0% to 41.6%.Without the editing command, agents commonly resorted to sed for simple single-line edits.Source: §4.2 · p. 9

4.3 Error Analysis

  • Agents performed best on specific error-resolution categories and worst on open-ended data, configuration, and goal problems.Accuracy was 27% for data, 38% for configuration, and 43% for goal, compared with 73% for CPU, 61% for issues, and 54% for dependencies.Source: §4.3 · p. 9
  • Trajectory analysis found that agents often skipped repository inspection, hallucinated script or function arguments, and failed to reconsider an approach after committing to it.These behaviors point to weaknesses in repository understanding and multi-approach problem solving.Source: §4.3 · pp. 9–10
  • The Script-Executed proxy agreed with landmark scores in 90% of cases and with accuracy in 69%, but it could misclassify late failures or delayed misconfigurations.The proxy requires the target script to run without exceptions for a minimum duration.Source: §4.3 · p. 10

5 Conclusion

SUPER introduces a benchmark for evaluating LLM-based agents on tasks from low-profile research repositories and shows that these tasks remain difficult. The benchmark highlights repository reasoning and code editing as core challenges for autonomous execution agents.

5 Conclusion

  • SUPER evaluates LLM-based agents on executing tasks from low-profile research repositories and finds the benchmark difficult even for current commercial models.GPT-4o solves only 46.1% of the sub-problems.Source: §5 · p. 10
  • The benchmark exposes core challenges in building autonomous execution agents, including repository reasoning and code editing.Source: §5 · p. 10
  • SUPER is intended to help the community make measurable progress on autonomous research-task execution.Source: §5 · p. 10

Limitations

The benchmark’s evaluation scope is limited by dataset size, repository language and domain coverage, and reliance on external resources. These choices trade broader control or diversity for accessibility and fidelity to realistic execution settings.

Dataset Size

  • The Expert and Masked sets contain 45 and 152 problems, which may affect the statistical significance of performance evaluations.The larger Auto set contains 604 development problems.Source: Limitations · p. 10

Programming Languages and Domains

  • SUPER supports only Python and focuses mostly on text-based repositories, leaving broader domain diversity for future work.Source: Limitations · p. 10

Evaluation Based on External Resources

  • Evaluation depends on external resources such as GitHub, pip, and datasets that the authors cannot control across runs.The benchmark prioritizes fidelity to realistic environments over complete sandboxing.Source: Limitations · p. 10

Ethical Considerations

Autonomous research execution agents may accelerate research, but careless reliance on their outputs could produce conclusions based on incorrect implementations.

Ethical Considerations

  • Over-reliance on autonomous agents could lead to conclusions based on incorrect implementations when reproduction work is not checked carefully.Source: Ethical Considerations · p. 11

A Repository details

The repository details report the 45 source repositories used for the Expert and Masked sets, including GitHub links and star information. The accompanying comparison summarizes average and median star ratings for related benchmarks.

A Repository details

B.1 Tasks Generation

SUPER generates tasks by filtering research repositories for feasible experiment setups, then prompting GPT-4o to create executable experiment-running tasks.

Step 1: Filtering Repositories

  • The pipeline begins with 5915 recent text-modality repositories and removes clones that fail, use LLM APIs, or lack README files.Repositories are drawn from paperswithcode entries added in 2021 or later.Source: §B.1 · p. 15
  • 1006 repositories remain after conservative filtering for supported datasets, CPU execution, model families, and executable examples.Repositories are filtered using GPT-4o predictions for these criteria, although additional high-quality repositories may have been excluded.Source: §B.1 · p. 15

Step 2: Generating Tasks

  • GPT-4o generates one task per filtered repository by selecting a README-documented experiment script and specifying its file path as the entrypoint.The prompt requires an executable Python or Bash file and skips repositories without a suitable example.Source: §B.1 · p. 15
  • Generated tasks must state the repository goal, name a specific supported dataset, identify the entrypoint, and select a small model from an approved family.The instructions favor small or base models such as bert-base, t5-small, gpt2, and pythia-70m to keep execution feasible.Source: §B.1 · pp. 15–16
  • Tasks are discarded when the model skips them, the selected script is missing, or the script is neither Python nor Bash.This final validation removes malformed or non-executable generated tasks.Source: §B.1 · p. 16

B.2 Tasks Evaluation (Script-Executed metric)

The script-executed metric approximates success for automatically generated tasks by checking exception-free execution for at least 10 seconds.

Script-Executed metric

  • 10 seconds is the selected runtime threshold for distinguishing successful script execution from quick failures without exceptions.The threshold was chosen as a trade-off based on gold expert solutions.Source: §B.2 · p. 16
  • The heuristic checks whether the target script runs without printed exceptions and continues for at least the threshold duration.It is intended for automatically generated tasks that lack gold solutions.Source: §B.2 · p. 16
  • The metric is approximate and may be incorrect or manipulable, but it matched landmark evaluation in 90% of cases.The authors present it as useful for development and potentially larger task sets.Source: §B.2 · p. 16

C Prompts and Interaction History

SUPER evaluates interactive agents through prompts that combine repository exploration, execution, editing, submission, and interaction history. The appendix specifies distinct prompt formats for ReAct-SUPER, Reflexion, and SWE-Agent variants.

C.1 ReAct-SUPER

  • ReAct-SUPER agents alternate one thought with one action, using execute, edit, or submit operations in a Jupyter interface.Execute supports Bash, Python, and magic commands; edit replaces exact file lines; submit returns the final answer or null.Source: §C.1 · p. 16
  • ReAct-SUPER instructs agents to clone repositories, read README instructions, install dependencies with pip, and manage five-minute execution timeouts.Agents should interrupt interactive prompts or clearly incorrect long-running executions, but continue correct runs.Source: §C.1 · p. 17
  • ReAct-SUPER examples demonstrate repository cloning, README inspection, package installation, evaluation execution, metric extraction, and final JSON submission.The demonstrations are separate from the SUPER sets and illustrate available actions.Source: §C.1 · p. 17
  • Interaction histories concatenate thought, action, and observation tuples, while masked tasks preload user-executed steps and truncate older observations.The last step retains up to 50k characters, whereas earlier observations are shortened to their last 500 characters.Source: §C.1 · pp. 18–19

C.2 Reflection Agent Prompt

  • The Reflexion prompt asks unsuccessful agents to identify strategic mistakes and produce a concise, environment-specific plan for retrying the same task.It emphasizes correcting omitted actions and avoiding inefficient paths.Source: §C.2 · p. 19

C.3 SWE-Agent Prompt

  • The SWE-Agent prompt adapts software-engineering interaction to research tasks while preserving comparable tips and demonstrations for fair comparison.Its format requires discussion plus one command and allows repository editing, checks, and tests.Source: §C.3 · pp. 19–20
  • SWE-Agent must issue only one command at a time and inspect files, directories, indentation, and edits before continuing.The prompt also recommends cloning repositories, reading usage instructions, installing packages, and checking failed commands.Source: §C.3 · p. 20
  • The appendix exposes a notebook-style environment with file-window context, Bash/Python/magic commands, navigation tools, and explicit indentation requirements.Execution timeout and interruption guidance parallels the ReAct-SUPER setup.Source: §C.3 · p. 20

E Edit command

The edit command replaces exact file content between BEFORE_EDIT and AFTER_EDIT markers, with feedback for whitespace, ambiguity, or unmatched lines.

E Edit command

  • The edit action replaces specified lines in a named file with new lines between BEFORE_EDIT and AFTER_EDIT markers.Agents must provide the exact existing lines followed by their replacements.Source: E Edit command · p. 21
  • Agents typically inspect files and copy the target content verbatim before issuing an edit.The environment searches for the supplied content and performs the replacement.Source: E Edit command · pp. 21–22
  • Edit inputs must match precisely, including whitespace and indentation, and must occur uniquely in the file.The environment reports missing matches or multiple occurrences to help agents disambiguate edits.Source: E Edit command · pp. 22–23
  • When multiple matches occur, agents are asked to add one to three surrounding lines to identify the intended occurrence.The feedback displays candidate occurrences with additional context.Source: E Edit command · p. 23
Loading 2409.07440v1…