Source-linked AI summary
EurekAgent: Agent Environment Engineering is All You Need For Autonomous Scientific Discovery
Amy Xin, Jiening Siow, Junjie Wang, Zijun Yao, Fanjin Zhang, Jian Song, Lei Hou, Juanzi Li
TL;DR
Reliable autonomous scientific discovery requires more than task performance because agents can compromise evaluation integrity and reproducibility. EurekAgent addresses this by engineering the agent environment across permissions, artifacts, budgets, and human oversight, achieving state-of-the-art results across mathematics and kernel engineering tasks and ranking first on evaluated MLE-Bench Lite.
Problem
Autonomous scientific discovery lacks reliable environmental controls against evaluation tampering, artifact manipulation, reward hacking, and observability failures.
Method
EurekAgent engineers permissions, artifacts, budgets, and human oversight while allowing CLI agents to choose their own research workflows and strategies.
Results
EurekAgent achieves new state-of-the-art results across all evaluated mathematics and kernel engineering tasks, ranks first on evaluated MLE-Bench Lite, and averages below $17 in API cost across three mathematics tasks.
Takeaways & Limitations
The paper presents environment engineering as a central layer for building capable, efficient, and responsible autonomous research agents.
Takeaways & Limitations
Current experiments focus on metric-driven tasks with executable evaluators, leaving broader and more open-ended scientific settings for future work.
Abstract
from arXiv · showhide
LLM-based agents have shown increasing potential in automating scientific discovery. Given an optimizable metric and an execution environment, they can propose, validate, and iterate scientific solutions, and have produced results that outperform human-designed approaches. As model capabilities continue to improve, we argue that the bottleneck for autonomous scientific discovery is shifting from prescribing agent workflows to designing agent environments: the resources, constraints, and interfaces that shape agent behavior. We frame this as environment engineering: building environments that amplify productive behaviors, such as open-ended exploration, systematic artifact management, and inter-agent collaboration, while suppressing harmful behaviors, such as reward hacking and high-friction human oversight. We present EurekAgent, an environment-engineered agent system for metric-driven autonomous scientific discovery. EurekAgent engineers the environment along four dimensions: permissions engineering for bounded agent execution and isolated evaluation; artifact engineering for filesystem and Git-based collaboration; budget engineering for budget-aware exploration; and human-in-the-loop engineering for easy human supervision and intervention. EurekAgent sets new state-of-the-art results on multiple mathematics, kernel engineering, and machine learning tasks, including new state-of-the-art 26-circle packing results discovered with less than $11 in total API cost. We open-source our code and results, and call for environment engineering as a core research direction for developing reliable autonomous research agents.
1 INTRODUCTION
The introduction argues that reliable autonomous scientific discovery increasingly depends on engineering the agent’s environment rather than prescribing detailed workflows. It presents EUREKAGENT, which applies permissions, artifact, budget, and human-in-the-loop engineering and achieves strong results across mathematics, kernel engineering, and machine learning tasks.
- Problem: Reliable autonomous researchers require rigor, reproducibility, and inspectability because agents can contaminate evaluations, manipulate artifacts, and violate procedural constraints.The introduction identifies reward hacking and observability failures as reported risks in agentic research systems.
- Motivation: As agents become more capable, the bottleneck shifts from prescribing agent behavior through detailed workflows to engineering the environments in which they operate.The paper frames this shift as environment engineering, emphasizing that environments shape available actions for good or ill.
- Contribution: EUREKAGENT coordinates off-the-shelf CLI agents through permissions, artifact, budget, and human-in-the-loop engineering dimensions.These dimensions respectively address bounded capabilities, shared progress memory, budget-aware exploration with runtime and compute boundaries, and supervision or intervention.
- Results: EUREKAGENT achieves new state-of-the-art results across all evaluated mathematics and kernel engineering tasks and ranks first on the evaluated MLE-Bench subset.The evaluation spans metric-driven research tasks in mathematics, kernel engineering, and machine learning engineering.
2 RELATED WORK
Prior work has pursued end-to-end autonomous scientific research and metric-driven discovery, while newer systems increasingly treat environment reliability and safeguards as central to agent reliability.
- Autonomous research agents: Autonomous research agents aim to accelerate scientific exploration through large-scale computational search.The AI Scientist targets end-to-end automation spanning idea generation, experimentation, and paper writing.
- Metric-driven discovery: Scientific discovery with verifiable objectives and optimizable metrics enables agents to autonomously explore and evolve solutions.
- Environment reliability: Environment reliability is increasingly recognized as a determinant of agent reliability, motivating safeguards such as leakage checking and hidden grader code.MLE-STAR adds leakage checking for machine learning pipelines, while CORAL hides grader code behind an evaluation interface.
3 EUREKAGENT
EUREKAGENT is an environment-engineered system that coordinates CLI-agent sessions to autonomously propose and iterate solutions for metric-driven research tasks. Its design uses bounded permissions, shared artifacts, budget controls, and human-supervision interfaces to make optimization productive, inspectable, and resource-bounded.
- System overview: EUREKAGENT accepts task inputs, evaluation specifications, optional initial code, and time/API budgets, then coordinates multiple CLI-agent sessions to pursue high-scoring solutions.The environment organizes agent activity without prescribing a detailed research workflow.
- System loop: Each iteration round contains one proposal session followed by up to P parallel implementation sessions, with R rounds and P sessions per stage configurable by the user.The outer loop initializes workspaces, assigns session objectives and deliverables, and manages tools and resources.
- Implement stage: The implement stage launches separate parallel sessions with separate workspaces, allowing agents to refine, debug, or modify hypotheses using hidden-evaluator feedback.Candidate solutions are submitted through a secure evaluation service that records evaluated submissions.
- Environment engineering: EUREKAGENT engineers four dimensions—permissions, artifacts, budgets, and human-in-the-loop interfaces—to support open-ended optimization while preserving reliability, inspectability, and resource bounds.Budget settings control wall-clock time and API cost, while terminal and web interfaces support oversight and communication with active sessions.
- Artifact engineering: Filesystem contents and Git history serve as shared long-term memory for preparation summaries, proposals, hypotheses, solution code, evaluator feedback, and scored submissions.System-managed artifacts also log web-search history and automatically record and rank official scores.
4 EXPERIMENTS
EUREKAGENT is evaluated on optimizable-metric tasks across mathematics, kernel engineering, and machine learning engineering. It achieves state-of-the-art or competitive results while using training-free, environment-engineered autonomous iteration.
- Evaluation setup: EUREKAGENT is evaluated across mathematics, kernel engineering, and machine learning engineering using Claude Code, GLM-5.1, web search, and browser-navigation MCPs.The experiments focus on tasks whose progress is measurable through objective scores.
- Mathematics: EUREKAGENT establishes new state-of-the-art results on all three mathematical optimization tasks while remaining training-free.The mathematical tasks include 26-circle packing and other optimization problems evaluated against prior test-time training systems.
- Kernel engineering: The best EUREKAGENT TriMul kernel improves over the strongest regraded leaderboard solution by about 4.3% and over TTT-Discover by about 10.8%.The comparison uses local evaluation on the same A100 GPU because the official GPUMODE leaderboard was closed.
- Kernel engineering: The top four EUREKAGENT TriMul solutions all achieve median runtimes below 2031 µs, indicating stable high-quality optimization rather than a single lucky candidate.All candidates were regraded under the same local protocol and correctness checks.
- Machine learning engineering: EUREKAGENT reaches an 85.71% any-medal rate on the selected seven-task MLE-Bench Lite subset with one run per task.It also attains the highest gold-medal rate among methods using non-commercial open models, while running with open-source GLM-5.1 against closed-model baselines.
5 CONCLUSION AND LIMITATIONS
EUREKAGENT demonstrates an environment-engineered approach to metric-driven autonomous scientific discovery by coordinating CLI-agent sessions and shaping evaluation, memory, resource, and oversight mechanisms. The paper positions environment engineering as a central layer for reliable autonomous research while acknowledging its current focus on executable-evaluator tasks and releasing the system for further development.
- Conclusion: EUREKAGENT coordinates off-the-shelf CLI-agent sessions through a prepare-propose-implement loop while shaping reliable evaluation, shared progress memory, resource boundaries, and human oversight.The system uses Claude Code as the CLI agent and GLM-5.1 as the base LLM, according to the passage.
- Limitations and future work: Environment engineering is presented as a central layer for autonomous research, defining reliable feedback, persistent memory, resource control, evaluator integrity, human oversight, and recoverable long-running operation.The current experiments focus on metric-driven tasks with executable evaluators.
- Conclusion: EUREKAGENT is open-sourced as an initial platform for community contributions, richer research domains, and continued updates on its performance, capabilities, and boundaries.The authors describe it as a practical starting point for collective exploration of environment engineering as a foundation for reliable autonomous scientific discovery.
A EUREKAGENT HYPERPARAMETER SETTINGS
The experiments use a defined set of EUREKAGENT hyperparameters summarized in Table 5. Two key controls are R, the maximum propose–implement iteration rounds, and P, the maximum parallel implementation sessions per implement stage.
- Iteration control: R denotes the maximum number of propose–implement iteration rounds.R controls how many iteration rounds the agent may perform.
- Parallel implementation: P denotes the maximum number of parallel implementation sessions spawned in each implement stage.P controls parallelism during implementation stages.
- Hyperparameter overview: Table 5 summarizes the EUREKAGENT hyperparameters used in the experiments.The table is explicitly identified as the source of the experimental hyperparameter settings.
B SELECTED MLE-BENCH LITE COMPETITIONS
The study selects seven MLE-Bench Lite competitions across three difficulty tiers, using prior public leaderboard agents’ aggregate medal rates as a proxy for task difficulty.
- Easy: Easy competitions are histopathologic-cancer-detection (57.0%) and plant-pathology-2020-fgvc7 (49.7%).Both competitions have prior-agent aggregate medal rates above 40%.
- Medium: Medium competitions are aerial-cactus-identification (26.2%) and the-icml-2013-whale-redux (23.5%).Both competitions fall within the 15%–40% medal-rate tier.
- Hard: Hard competitions are jigsaw-toxic-comment (9.1%), dog-breed-identification (0.4%), and tabular-playground-may-2022 (0.4%).All three competitions have prior-agent aggregate medal rates below 10%.