Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

1 Introduction

• Open-ended discovery lacks clear progress feedback, leaving agents to judge intermediate results and risking divergence from researchers’ judgments or preference for easier questions.The introduction describes this preference for ease over scientific value as the streetlight effect [6–8].Source: §1 · p. 1
• Supervisor guidance and periodic Meta Reflection are designed to keep agents committed to proposed directions and reassessing whether those directions align with human researchers’ interests.The authors say the mechanisms together encourage persistence in scientifically valuable directions rather than premature pivots to easier questions.Source: §1 · p. 2
• 62.7% of withheld sub-discoveries were recovered by extended Station on average, compared with 15.4% for Codex Multiagent-v2 and 14.4–20.6% for AI Scientist-v2.The three tasks draw their primary research questions from recent oral papers at ICLR; agents received the research questions but not the papers’ findings, and web access was disabled.Source: §1 · p. 2
• On two additional tasks without oracle papers, agents produced findings closely aligned with those in concurrent human-research papers released after the agents’ knowledge cutoffs.For subliminal learning, agents also proposed an explanation for failed trait transfer and a method for restoring it.Source: §1 · p. 3
• The authors identify a continuing gap from human research: agents miss findings requiring sustained, cumulative work and spend effort on directions human researchers find unpromising.They also release the source code and full agent dialogues for further research.Source: §1 · p. 3
• The study examines autonomous discovery across five open-ended tasks using comparative evaluations, ablations, and behavioral analyses to assess system capabilities and limitations.The paper adapts Station from metric-based tasks and introduces the Supervisor and Meta Reflection mechanisms.Source: §1 · p. 3

2 Related Work

• Recent case studies report unreliable novelty assessments and poor research judgment, while Kosmos and Google’s Co-Scientist extension report rediscoveries or new findings on open-ended tasks.The passage notes that both Kosmos and the Co-Scientist extension are closed-source.Source: §2 · p. 3
• Open-source systems AI Scientist-v2 and Codex Multiagent-v2 organize research through agentic tree search or a root agent delegating work to sub-agents.The passage characterizes these approaches as predefined pipelines or central coordination.Source: §2 · p. 3
• Unlike those centralized approaches, Station lets agents independently propose and pursue research directions, and this paper extends it beyond tasks with explicit evaluation metrics.Source: §2 · p. 3

3 Problem Setting and Background

• Open-ended tasks lack a well-defined objective metric for assessing progress or the scientific value of results, so judging scientific merit often requires researchers’ subjective expertise.The passage contrasts a complete description of an interpretability circuit with a more scientifically valuable explanation of how it implements a function.Source: §3 · p. 3
• Station simulates a decentralized scientific ecosystem in which agents with different models and contexts independently choose directions, run experiments, and communicate.Source: §3 · p. 3
• Station maintains a persistent internal literature system for agents to publish, read, and build on findings, and runs in discrete ticks with observations and responses from active agents.Typically, six agents are active.Source: §3 · p. 4

Supervisor.

• The Supervisor requires agents to propose a research direction and commit to it for a fixed period, monitoring progress while allowing exceptional pivots.The Supervisor cannot submit experiments or publish papers; its role is limited to guidance and oversight.Source: Supervisor · p. 4

Meta Reflection.

• Every 50 ticks, Meta Reflection prompts agents to reassess their progress, research direction, and alignment with human researchers’ judgments of scientific value.Responses use GPT-5.5 regardless of the agent’s underlying model.Source: Meta Reflection · p. 4
• A final Consolidation Agent integrates findings and supporting evidence into an approximately 8,000-word research report, replacing selection by an explicit evaluation metric.The agent is implemented using Codex with GPT-5.5.Source: Consolidation Agent · p. 4

• Three rediscovery tasks use research questions from oral papers at ICLR 2025 and ICLR 2026, with oracle-paper findings withheld; two exploratory tasks have no designated oracle paper.Exploratory findings are assessed through human evaluation.Source: §5 · p. 4
• Each rediscovery task has three independent runs; each exploratory task has one, with six agents running for 300 ticks without internet access or human intervention.The six agents comprise three Gemini 3.1 Pro agents, one GPT-5.5 agent, and two Claude Opus 4.8 agents.Source: §5 · pp. 4–5
• Rediscovery scores measure the share of approximately ten oracle-paper sub-discoveries that reports state substantively and support with evidence.Each run’s score averages nine automated evaluations across three reports, and blinded expert assessment shows high agreement with those evaluations.Source: §5 · p. 5
• Exploratory findings are manually screened for interest, compared with related work, and validated when promising; reported findings are novel or coincide with concurrent work.The agent dialogues and reports are publicly available for independent verification.Source: §5 · p. 5
• Baseline systems receive the same research questions and setups as Station, matched cumulative experiment time, and outputs of at least 8,000 words assessed by the same procedure.The baselines are AI Scientist-v2 with Claude Opus 4.8, Gemini 3.1 Pro, or GPT-5.5, and Codex Multiagent-v2 with GPT.Source: §5 · p. 5
• Figure 2 reports sub-discovery recovery by category and whole task for three rediscovery tasks, with means and minimum–maximum ranges across three runs.Baseline cumulative experiment time is matched to Station’s.Source: Figure 2 · p. 6

Emergent Planning in RL Agents.

• 75.8% (8.33/11): Station recovered more emergent-planning sub-discoveries than Codex Multiagent-v2 (6.1%) and AI Scientist-v2 (6.1–19.2%), but struggled with causal interventions and deeper explanations.Agents often probed concepts without using them to explain bidirectional planning or to manipulate behavior.Source: Emergent Planning in RL Agents · p. 6

Low-Rank Structure in LLM Outputs.

• 70.4% (8.44/12): Station nearly recovered sequence-level structure and reusable relations, outperforming Codex Multiagent-v2 (33.3%) and AI Scientist-v2 (32.4–37.0%), but recovered few functional consequences.Research records suggest Station agents often pursued deeper questions, while baselines largely stayed with characterizing low-rank structure.Source: Low-Rank Structure in LLM Outputs · pp. 6–7

Temporal Representations in RNNs.

• 41.9% (4.19/10): Station recovered linear dynamics and spatial–temporal tradeoffs more strongly than nonlinear dynamics, and outperformed Codex Multiagent-v2 (6.7%) and AI Scientist-v2 (0.0–11.1%).Agents often optimized memory performance under limited capacity rather than investigating temporal-representation and forgetting mechanisms.Source: Temporal Representations in RNNs · p. 7

6.1 Effects of the Supervisor and Meta Reflection

• 75.8% to 57.9%: Removing Supervisor and Meta Reflection reduced emergent-planning recovery; research projects also shortened from 67.0 to 55.5 ticks, and 71.7% of reflection events were followed by experiments.Ablated agents were less likely to build on earlier discoveries, while Meta Reflection could redirect an analysis toward the central planning question.Source: §6.1 · p. 7, Figure 3 · p. 8

6.2 Gaps Between AI Agents and Human Researchers

• 75% of 24 proposals received 1–2/5; agents also favored easier performance-optimization questions over mechanistic explanations, leaving the streetlight effect unresolved.The author-rated proposals focused on minor details judged to offer little insight into the main question.Source: §6.2 · pp. 7–8

7 Exploratory Tasks

• Agents found an inverted-U relation between subliminal-learning transfer and LoRA rank; a few leading early-layer update modes retained most of the trait, and restricting rank-128 adapters to early layers restored transfer.The early-layer result went beyond localization findings reported by Nief et al. [26].Source: §7 · pp. 8–9, Figure 4 · p. 9

VLM Hallucination.

• Scaling the MLP outputs of the final two layers by 0.5 during prompt prefill reduced content-based hallucinations; authors propose this attenuates language priors that can override visual evidence, aligning with concurrent work by Guo et al. [27].Source: VLM Hallucination · p. 9

8 Discussion and Conclusion

• Station’s decentralized environment, Supervisor guidance, and Meta Reflection support sustained exploration and reassessment, and agents recovered findings and made new discoveries on open-ended tasks.The authors leave the individual contributions of the environment’s components open.Source: §8 · p. 9
• Judgment misalignment and the streetlight effect remain unresolved; human steering could help narrow the gap but would limit the scalability of autonomous research.Improved research-environment design remains an important direction.Source: §8 · p. 9
• The figure reports that scaling layers 34 and 35 MLP outputs by 0.5 during prompt prefill raised correct answers from 28 to 35 on a 40-example visual-hallucination dataset.The authors also report improved performance on additional models and a larger evaluation set.Source: Figure 5 · p. 10

A.1 Space, Time, and Action

• Station is a discrete-time environment with modular rooms and actions; agents receive observations and respond each tick, while web access is disabled and no task-level score is provided.The Research Center supplies experimental materials and a sandbox for arbitrary code execution.Source: §A.1 · p. 14, §A.3.1 · p. 15

A.2 Agents

• The six-agent composition is maintained as agents leave and are replaced; lineages preserve research identity and private notes, while roles support diverse research styles and agents have a 150-tick maximum lifetime.Agents work in isolation for their first 20 ticks before gaining access to collaborative rooms.Source: §A.2 · pp. 14–15

A.3.1 Research Center; A.3.2 Archive Room

• The Research Center supports experiment execution and storage, while the Archive Room accepts reviewed papers that later agents can read and build upon.Archive submissions should synthesize experiments into a scientific account rather than merely list a research log.Source: §A.3.1 · p. 15, §A.3.2 · p. 15

A.3.3 Mail Room; A.3.4 Public Memory Room; A.3.5 Common Room

• The Mail Room enables private peer communication, the Public Memory Room preserves public threads, and the Common Room supports live group conversation.Public Memory threads persist after authors leave, unlike the Common Room’s recent-message chat.Source: §A.3.3 · p. 16, §A.3.4 · p. 16, §A.3.5 · p. 16

A.3.6 Reflection Chamber

• The Reflection Chamber supports uninterrupted reflection sessions, including periodic Meta Reflection prompts sampled from a Station-supplied pool.A reflection session occurs within one tick as a separate dialogue during an agent’s turn.Source: §A.3.6 · pp. 16–17

B.1 Supervisor

• A GPT-model agent becomes Supervisor after publishing a paper when none is assigned, then reviews proposals and meets with agents while preserving their autonomy.Once assigned, the Supervisor can no longer submit experiments or papers.Source: §B.1 · p. 18

B.1.1 Supervisor system prompt

• The Supervisor system prompt enforces a minimum 60-tick commitment, tracks approved directions, and rejects premature pivots while forbidding it from steering agents’ research.Regular one-to-one meetings occur every 10 ticks, or every 20 ticks for tenured agents.Source: §B.1.1 · pp. 18–19

B.2 Meta Reflection

• Mature agents must perform Meta Reflection in the Reflection Chamber every 50 ticks; missed reflections trigger a system reminder.Each reflection uses a sampled prompt followed by two continuation prompts, with the model temporarily switched to GPT-5.5 for all three turns.Source: §B.2 · p. 21
• Meta Reflection prompts cast the agent’s past work as an external review, asking it to examine overclaiming and whether it has a persistent long-term research plan.The sampled prompts include scrutiny of experimental interpretation and method investigation, and a request to design a 60-tick plan.Source: §B.2 · p. 21
• Meta Reflection prompts agents to assess novelty and diversity, brainstorm motivated ideas, challenge shared assumptions, and record and pursue promising ideas.It also suggests technical innovations within existing method classes, not only entirely new methods.Source: §B.2 · p. 22
• Agents review whether their work resembles expert human research and whether their direction is too rigid or volatile, then update their prompts or save a pivot policy.The pivot review considers experiments from the last 50 ticks and asks agents to balance reckless pivots against deadlock in one method class.Source: §B.2 · pp. 22–23
• To counter attractor tasks, agents identify futile closure, meta-work, or minor SOTA variations and pursue high-risk ideas, including risky hypotheses, cross-field transfers, and creative constructions.The reflection favors novel paths within the current lane, reduces time spent on attractors, and asks agents to try proposed paths.Source: §B.2 · p. 23
• Meta Reflection may have agents select a simpler, related, or seemingly unrelated subproblem to unblock the main task, while checking for reward hacking and genuine scientific value.Agents are asked to compare candidate subproblems and may focus on one temporarily; reward-hacking checks include shallow shortcuts, trivial publications, and flawed impossibility claims.Source: §B.2 · pp. 23–24
• Agents are asked to adopt and preserve a distinct research taste and crystallize knowledge gained from experiments and Station papers in Private Memory Room.The prompts distinguish knowledge learned in Station from pretrained knowledge.Source: §B.2 · p. 24

B.3 Consolidation Agent

• The Consolidation Agent must synthesize archived papers, Station analyses, and experimental evidence into a self-contained research paper of at least 8,000 words.It should follow the task specification, integrate a coherent scientific story rather than list logs or summarize papers, and base factual claims on supplied records.Source: §B.3 · pp. 24–25
• The Consolidation Agent is restricted to the supplied research-record directory and must write its final paper to a run-specific output directory.It may inspect records and run read-only commands or analysis scripts, but may not modify records, run experiments, browse the web, or consult outside sources.Source: §B.3 · p. 25
• The agent should prioritize important results, distinguish uncertain claims from supported findings, and present the strongest findings without unnecessary hedging.Negative results or minor observations may be omitted unless important to the main story.Source: §B.3 · pp. 25–26
• Writing should be clear and engaging, define task-specific terms, address an expert unfamiliar with Station, and use Markdown and headings appropriately.Tables are optional where helpful.Source: §B.3 · p. 26
• The agent should plan and revise a scientifically integrated narrative that fits the findings rather than forcing a generic template.The paper should read as one story, not as a concatenation of summaries, experiment notes, or isolated findings.Source: §B.3 · p. 26
• Before finishing, the agent should verify that the standalone paper answers the research questions, reaches at least 8,000 words, and explains Station references for external readers.The checklist also requires that the research-record directory remain unchanged.Source: §B.3 · p. 26
• All final outputs should be stored under the run-specific FINAL_OUTPUT_DIR.The required paper path is FINAL_OUTPUT_DIR/research_paper.md.Source: §B.3 · p. 26

C.1 Emergent planning in RL agents

• The planning rubric tests whether future consequences are represented in agent states, predict multiple steps ahead, and can be recovered from ConvLSTM cell states with a spatially local probe.The criteria distinguish predictions of multiple future steps from predictions of only the next action.Source: §C.1 · p. 27
• Planning criteria also cover progressive refinement, revision of infeasible or inferior routes, bidirectional plan formation, and parallel extension of partial plans.The rubric describes forward extension from boxes and backward extension from targets as consistent with bidirectional search.Source: §C.1 · p. 27

C.2 Low-rank structure in LLM outputs

• The LLM-output rubric tests whether a sequence-level object is approximately low-rank and whether that structure persists as histories, continuation blocks, and analyzed entries change.Evidence may include singular-spectrum analysis, low-rank reconstruction, or held-out prediction.Source: §C.2 · p. 27
• Reusable-relation criteria require a history-side relation learned on one continuation set to transfer without refitting, with matched controls ruling out simpler explanations.The rubric also distinguishes functional transfer evidence from descriptive geometry alone.Source: §C.2 · p. 28
• Functional criteria test whether relations among histories predict target outputs across multiple autoregressive positions, with baselines or controls and target-specific information preserved.The rubric contrasts this evidence with prediction only at the first next-token position.Source: §C.2 · p. 28

C.3 Temporal representations in RNNs

• Linear-RNN criteria track age-dependent feature trajectories and gradual forgetting, alongside the tradeoff between representing more features and retaining them longer.The criteria describe ordered changes in activation space and progressive weakening of older, task-irrelevant contributions.Source: §C.3 · pp. 28–29
• Nonlinear-RNN criteria distinguish abrupt suppression of outdated information from gradual decay and assess whether nonlinear geometry explains that sharp forgetting.The rubric calls for direct evidence from hidden states, feature trajectories, or readouts; improved task loss alone is insufficient.Source: §C.3 · p. 28
• The RNN capacity rubric also tests regime changes, preferential retention of higher-utility features, full-window persistence, and whether greater dimensionality leaves the breadth-duration tradeoff intact.Partial, short-lived retention is not treated as an effective substitute for persistence across the task-relevant memory window.Source: §C.3 · p. 29

D Evaluation Details

• Across 66 blinded judgments, automated evaluations agreed with the author on 64 (97.0%).The author reviewed sampled outputs without seeing automated evaluations or system identities.Source: D Evaluation Details · p. 30
• The rubric scores each predefined claim 1 only when its scientific meaning matches, claim-specific evidence supports it, and the final paper does not contradict it.Evaluators build separate reference and final-paper evidence ledgers before assigning scores and auditing the report.Source: D Evaluation Details · p. 30

E.1 Codex Multiagent-v2

• Codex’s root agent received Station’s task specification and experimental materials, with a matched minimum cumulative experiment-time requirement.Its final synthesis was required to be approximately 8,000 words and link substantive conclusions to evidence.Source: §E.1 · pp. 32–33
• Codex Multiagent-v2 repeatedly selects questions, runs experiments or audits, synthesizes findings, and revises claims until research and runtime criteria are satisfied.Later rounds can reproduce conclusions, test generalization, strengthen controls, or investigate uncertainty and failure cases.Source: §E.1 · p. 32

E.2 AI Scientist-v2

• AI Scientist-v2 used its official four-stage workflow, adding tree-search nodes until cumulative experiment time met or exceeded Station’s task-specific average.Stages 1 and 3 received at least 100 nodes each, and Stages 2 and 4 at least 25 each.Source: §E.2 · p. 34
• AI Scientist-v2’s official write-up tool generated the final paper from the complete research record, with an 8,000-word minimum.Source: §E.2 · p. 34

E.3 Resource Usage and Detailed Rediscovery Results

• Station’s average experiment-time thresholds were approximately 44 hours for emergent planning, 18 hours for low-rank structure, and 64 hours for temporal representations.Each baseline ran until its cumulative experiment time met or exceeded the corresponding task threshold.Source: §E.3 · p. 34
• By tick 100, Station already recovers most findings obtained by tick 300, suggesting shorter runs may achieve similar rediscovery performance.Source: §E.3 · p. 34
• Tables 2–4 cover system token use and experiment time, checkpoint and baseline sub-discovery recovery, and equal-task-weighted average recovery across tasks.Source: Table 2 · p. 35, Table 3 · p. 36, Table 4 · p. 36

F Limitations

• Because model cutoffs span January 2025 to January 2026 and the earliest oracle paper appeared in April 2025, prior exposure to its findings cannot be ruled out.The authors note that baselines and direct prompting failed to recover most findings, while exploratory discoveries aligned with post-cutoff concurrent work.Source: F Limitations · p. 37
• The study cannot attribute Station’s performance gains specifically to decentralization because systems also differ in model composition and research workflow.Isolating individual design choices through controlled comparisons and separate ablations remains future work.Source: F Limitations · p. 37
• Rediscovery tasks do not comprehensively assess scientific value, and defining sub-discovery rubrics involves subjective judgment.The rubrics were defined before Station runs and made publicly available for independent assessment.Source: F Limitations · p. 37

G.1 Subliminal learning

• Early-layer LoRA fine-tuning increased target-animal preference across all five traits and both tested ranks, relative to fine-tuning across all layers.The results suggest early-layer adaptation is sufficient for trait transfer, while extending adaptation to later layers can weaken it.Source: §G.1 · p. 38, Table 5 · p. 38
• 47.63% mean cat preference came from adapting the first eight layers, versus 30.23% for the 14-layer default, a 17.40-percentage-point increase.This highest-performing cutoff was tested for the cat trait at rank 128; other traits and training settings remain untested.Source: §G.1 · p. 38, Table 6 · p. 38
• The observed inverted-U relation between LoRA rank and trait transfer may disappear with rank-specific learning-rate tuning and sufficient training data.How rank, layer selection, learning rate, and data availability jointly affect transfer remains unresolved.Source: §G.1 · p. 39

G.2 VLM hallucination

• On AMBER-1000, MLP-output attenuation improved accuracy across four VLMs by 0.80 to 13.70 percentage points, with effects varying by model.The external experiments suggest mild attenuation can reduce visual hallucinations across VLMs.Source: §G.2 · p. 39, Table 7 · p. 39
• For InternVL3.5-8B, stronger MLP-output attenuation began to hurt performance, while a 0.75 multiplier performed best among tested values.Source: §G.2 · p. 39, Table 8 · p. 39