PaperlayerView exact revision on arXiv

Source-linked AI summary

Letting AI Formulate the Research Question: Autonomous Research Agents

Ko Fujimura

arXiv:2610.39620v1cs.AIcs.CY

TL;DR

Earlier demonstrations left people to choose the research area and which outputs to show, leaving the effect of AI-formulated questions and unfiltered reporting to study. This paper gives a coding agent instructions and two skills to formulate questions and review its work, then reports five consecutive runs. The runs completed through revised papers, while the authors identify question choice as the system’s weakest point.

  • Problem

    Earlier systems left people to choose the research area and select which outputs to show, so this paper studies AI-formulated questions and reporting every run.

    Source: §1 · p. 1

  • Method

    A coding agent uses an instruction document and question-formulation and peer-review skills, with a different-model critique and a reviewer rerun from study records.

    Source: §1 · p. 2

  • Results

    All five runs completed question formulation through experiments, writing, AI review, and revision without research advice or permission requests from people.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The authors conclude that formulating questions that matter is where autonomous research agents now need the most work.

    Source: §9 · p. 15

    The evaluation comprised five runs with one research model and no ablation, all evaluations were by AI models, and human evaluation remains planned.

    Source: §8 · p. 15

Abstract

from arXiv · show

Systems such as The AI Scientist have shown that AI agents can carry out and write up research, but in those demonstrations humans chose the research area and selected the outputs to show. We study letting the AI formulate the research question itself, and we report every run. Our system gives an off-the-shelf coding agent an instruction document and two skills: one for formulating the question, in which the agent diverges from recent technologies into problems of other fields, screens its candidates for known failure modes and obtains a critique from a different model; and one for peer review, in which a reviewer from another model family reruns the study from its records before the agent revises the paper. Over 13 development runs, changes to the one or two sentences that set the research area coincided with large shifts in what the candidate questions and the chosen papers were about, which led us to the area of a new technology applied to a societal problem. We froze the system after a pilot run and report five consecutive runs: the pilot, counted by a disclosed amendment to the pre-registration, and four pre-registered runs. All five runs went from question formulation to a revised, re-computable paper, with local computation on a laptop and no research input from humans, and external AI reviewers reading only the papers judged them at or near workshop level. We release the system, the five papers and the records needed to reproduce them.

1 Introduction

The paper shifts research-area setting, question selection, and output selection from people to an AI agent, while reporting every run. It presents a plain agent-and-skills system and five consecutive runs, alongside lessons from 13 development runs and cross-model critique and review.

1 Introduction

  • The system leaves the specific question and reported outputs to the AI while people set the area broadly and report every run.The human sets the area in one or two sentences, here societal problems that can be studied with open data.Source: §1 · pp. 1–2
  • Its question-formulation skill screens candidates for definition-based answers, decorative technology, and lack of public ground truth, then obtains a critique from another model.A reviewer from another model family reruns the study from its records before the agent revises the paper.Source: §1 · p. 2
  • All five runs completed question formulation, experiments, writing, AI review, and revision locally on a laptop, without research advice or permission requests from people.The pilot was counted through a disclosed amendment; four runs were pre-registered.Source: §1 · p. 2
  • Across 13 development runs, changes to the one or two sentences setting the research area coincided with large shifts in candidate questions and chosen papers.The authors describe the path to a setting of new technology applied to societal problems.Source: §1 · p. 2
  • The critic changed comparisons and measurements in every run, while the reviewer recomputed every first draft and identified interpretation problems the agent fixed.The paper also reports external AI evaluation of the five final papers by reviewers from two further model families.Source: §1 · p. 2

2 Related work

Prior systems automate research pipelines, parts of research, or discovery in particular sciences, but often retain human-set goals or inputs. This paper distinguishes its approach by broadly setting the area, leaving question and output selection to the AI, and retaining records for recomputation.

2 Related work

  • The AI Scientist [6] introduced a pipeline from idea generation through experiments, write-up, and automated review; The AI Scientist-v2 [19] added template-free experimentation and tree search.Its journal version reports improved paper quality according to its automated reviewer with newer foundation models and more test-time compute.Source: §2 · p. 2
  • Agent Laboratory [13] and AgentRxiv [12] use agents as research assistants with human input, while Jr. AI Scientist [10] extends a baseline paper and reports observed risks.Kosmos [9] and Denario [18] target data-driven discovery; the AI co-scientist [5] generates and ranks hypotheses for scientists.Source: §2 · p. 2
  • AI-Researcher [16] includes open-ended tasks using reference papers and datasets; this system instead sets the area broadly, filters no outputs, and includes recomputation records.Prior work reports that workflow logs and code reveal some research failures more effectively than final papers alone [8].Source: §2 · p. 3
  • Research-idea studies report that LLM ideas were judged more novel but slightly less feasible than experts’ ideas [14], with the advantage disappearing upon execution [15].This paper observes question formulation within complete, unfiltered runs and the critiques and reviews that follow.Source: §2 · p. 3

3 System

The system combines a human-managed run, a research agent, a critic, and a reviewer with instructions, two skills, and append-only records. The question procedure directs candidate generation and screening; records support reviewer reruns and regeneration of reported outputs.

3.1 Roles

  • A run involves a human and four agents, with the management agent starting and resuming the research session and relaying the reviewer’s report.The management agent sends no research advice, and its non-instruction writing is not visible to the research session.Source: §3.1 · p. 4
  • The research agent formulates the question, conducts the study, and writes the paper; a critic reviews its top five candidates, and a reviewer reruns the study from the draft and records.The reviewer checks that plans preceded results and gives the research agent two hours to respond.Source: §3.1 · p. 4

3.2 Instructions and skills

  • The instruction document sets resource and safety conditions, says negative results and failures should be submitted, and hints at societal problems studied with open data.It does not name a field or topic; the checks and scores are the agent’s own judgments, and nothing enforces them.Source: §3.2 · pp. 4–5
  • The research-question skill starts with obvious candidates, then diverges from recent technologies toward questions in other fields and checks for public data.It scores candidates across seven axes and selects using impact, novelty, non-obviousness, generality, and unpredictability; feasibility scores of two or less trigger narrowing and rescoring.Source: §3.2 · pp. 4–5
  • Candidates are screened for decorative technology, prior application, definition-based results, public ground truth, usability, and whether answers change a decision.The separate peer-review skill tells the reviewer to rerun verification without AI API calls and check claims, plans, and prior work.Source: §3.2 · p. 5

3.3 Run records and operation

  • The append-only records include plans, event and input timelines, claim evidence, API calls and costs, environment manifests, version tags, and file hashes.A verification entry point regenerates reported numbers, tables, and figures without AI calls; computation ran locally on a MacBook Pro with 18 GiB memory and no discrete GPU.Source: §3.3 · p. 5

4 Setting the research area: lessons from 13 development runs

Across 13 development runs, changes to the instructions’ research-area setting coincided with shifts in candidate themes and chosen studies, though the observations were not controlled comparisons. The path to technology applied to societal problems combined a domain hint with technology-first question generation and additional checks.

No area

  • The first run, with no area or selection procedure, listed two candidates, chose an exact comparison in a classical probability problem, and finished in about 12 minutes.The authors caution that development-run observations are not controlled comparisons because the model, environment, and other instructions also changed.Source: §4 · p. 6

Computer science

  • With computer science set as the area, chosen studies stayed in computer systems and AI tooling; run 010 had 7 of 15 candidates on compression, storage, caching, or deduplication.The authors suggest the agent appeared to derive topics from constraints imposed by its own working conditions.Source: §4 · p. 6

Single sentences and the agenda

  • Within computer science, instruction-sentence changes coincided with shifts in candidate themes, while near-unchanged area and selection instructions coincided with many repeated candidates.When the reader changed to researchers and practitioners, the next run wrote no AI or research-practice candidates, compared with four in the previous run of the same model.Source: §4 · p. 6

Societal problems

  • After adding a societal-problems hint, 14 of 15 candidates concerned societal problems, and the chosen study shifted from workflow caches to PFAS monitoring in drinking water.The authors state that they did not test removing the field without adding the hint, so this was not simply more freedom.Source: §4 · pp. 6, 8

Technology × societal problem

  • Technology-first divergence combined with the societal-problems hint; both research models then produced almost entirely societal candidates, but differed in country and subject.The resulting English evaluation instructions restricted starting technologies to capabilities first possible around 2024 or later and checked for prior application to the same problem.Source: §4 · p. 8

The checks

  • The added checks address decorative technology, unverifiable judgments, definition-driven findings, and low feasibility, while the authors caution that observed shifts were not isolated from simultaneous changes.They expect more specific settings may produce correspondingly specific themes, but have not tested this; the fixed evaluation setting also led agents toward a few well-documented resources.Source: §4 · p. 8

5 Evaluation design

The evaluation froze the system and defined a five-run, record-based protocol, but later amended the plan to count the pilot after seeing three later-run results. It also specifies the reporting instruments and external AI review design.

5 Evaluation design

  • Table 2 tracks topic-selection and draft timing, stops, active time, deadline margin, resumes, permissions, advice, review, response, and critique API cost.Times are rounded from seconds, so active time may differ from draft time minus stops by one minute.Source: Table 2 · p. 9
  • The frozen evaluation pre-registered runs 015–019 and an all-starts inclusion rule; after runs 015–017 were seen, an amendment counted the pilot and omitted run 019.The authors report this as a deviation; the pilot’s workspace also contained no earlier runs.Source: §5 · pp. 8–9
  • Run-level reporting covers times, stops, inputs, candidate fates, critique decisions, and review responses; a script generated all table numbers from the records.The records include both run and management records.Source: §5 · p. 9
  • Two reviewers from other model families each reviewed every final paper twice from the papers alone; these are uncalibrated AI assessments and repeated reviews of five papers.The reviewers were told the papers came from an autonomous AI research system and could search the web, but received no code, data, or records.Source: §5 · p. 9
  • Tables 3 and 4 present the five run questions and candidate/critique counts, including critique adoption and points on the selected candidate.Table 3 expands LCMAP, NLCD, and ECMWF; Table 4 notes that critique did not change which candidate was chosen.Source: Table 4 · p. 10, Table 3 · p. 10

6 Results

Across five runs, the agents formulated and reported societal or environmental research questions, and cross-model review identified interpretation problems that prompted revisions. External AI reviewers judged the papers at or near workshop level, while rating novelty, generality, and clarity low.

6.1 Completion and inputs

  • All five runs produced revised drafts after AI review, with no recorded research advice or permission requests; records show deadline errors after conversation compaction in three runs.Agents spent 6.5 to 11.7 active hours before drafting, and usage-limit waits added up to 6.3 hours in one run.Source: §6.1 · p. 10

6.2 Questions the agents formulated

  • Across five runs, agents selected public-data societal or environmental questions; their 77 candidates spanned nine fields and were discarded for access, ground-truth, tautology, or prior-work problems.AlphaEarth satellite embeddings appeared in all five runs and were used in four chosen studies.Source: §6.2 · p. 11

6.3 What the critique changed

  • In all five runs, the critic left the chosen candidate unchanged but revised comparisons or endpoints, from held-out errors to controls and label-masked diagnostics.For run 016, the reference changed from a classified map to farmers’ declarations; for run 018, the comparison tested embedding distances beyond public covariates.Source: §6.3 · pp. 11–12

6.4 In-loop review and revision

  • The reviewer reproduced each first draft’s reported numbers and gave verifiability a score of 5 in every run; its 40 findings concerned interpretation, and all were adopted or partly adopted.Four drafts received major revision and one minor revision; the final versions were not reviewed again.Source: §6.4 · p. 12

6.5 What the final papers report

  • Two main results were negative or null; the positive findings in runs 015 and 017 stated limits on transfer and on what their endpoints measure.In run 016, revisiting reduced error, but predicting no change was better still.Source: §6.5 · pp. 12–13, Table 6 · p. 13

6.6 External AI evaluation

  • Gemini judged all five papers workshop level; Muse judged three of ten reviews yes and seven borderline, with two reject verdicts.Both reviewers rated novelty, generality, and clarity low; they judged the main answer partly predictable in 19 of 20 reviews.Source: §6.6 · p. 13, Table 7 · p. 14

7 Discussion

The runs show that delegating question choice makes the research area explicit rather than removing human influence; the procedure and open-data conditions also shape the resulting agenda. Execution and review were completed, while question importance and framing remain weaknesses and generality across area settings is untested.

7 Discussion

  • Five runs formulated and carried through their own questions without human research input, making the human-set area explicit rather than eliminating its influence.The agents chose societal and environmental questions using public data; the procedure and model also shaped where choices converged.Source: §7 · p. 14
  • The critic shaped comparisons and claims, while the reviewer checked whether claims followed evidence; the weakest point remains choosing a consequential question.The paper cannot estimate either tool’s effect because it did not evaluate papers before and after their use.Source: §7 · p. 14
  • Execution did not prevent completion in these five runs, which chose questions partly for feasibility; remaining problems concerned framing and predictability.These included which estimate led the abstract and whether controls received their planned weight.Source: §7 · p. 15
  • All five evaluation runs used one area setting; other settings and whether more specific settings still yield non-obvious questions remain untested.A human evaluation of the five papers is planned separately.Source: §7 · p. 15

8 Limitations

The evaluation is small and narrow, and its design does not isolate the effects of the question procedure, critic, or reviewer. AI-only judgments, non-independent recomputation, and several disclosed deviations further bound what the results establish.

8 Limitations

  • Five runs with one research model and no ablation cannot establish which outcomes are attributable to the question procedure, critic, or reviewer.The area-setting observations also come from development runs where the model, environment, and other instructions changed.Source: §8 · p. 15
  • All evaluations were AI judgments, and the external reviewers saw only papers; their uncalibrated ratings do not reflect the records or human decisions.The in-loop reviewer shared the management agent’s model family.Source: §8 · p. 15
  • Reviewer recomputation reran the agent’s own code, so it does not establish reference-label correctness or interpretation validity.Isolation from earlier runs relied on separate configuration and instructions, not access control.Source: §8 · p. 15
  • The pilot was counted after three results were seen, and PDF reviews were chosen as the main external result after Markdown reviews were seen; both deviations are reported.Source: §8 · p. 15

9 Conclusion

A plain coding agent with an instruction document and two skills repeatedly formulated research questions and produced revised, verifiable papers without human research input. AI reviewers rated the papers favourably or borderline for workshops but below main-track level, with low novelty and generality; the system and reproduction records are released.

9 Conclusion

  • A coding agent with an instruction document and two skills formulated its own question and produced a revised, verifiable paper repeatedly without human research input.Source: §9 · p. 15
  • AI reviewers reading only the papers judged them favourably or borderline for workshops and below the main track, while rating novelty and generality low.The judgments were not calibrated against human decisions.Source: §9 · p. 15
  • Development runs showed that the research area shifted with the one or two sentences setting it, while question formation within that area depended on the procedure.The paper identifies formulating questions that matter as the area where autonomous research agents need the most work.Source: §9 · p. 15
  • The system, five papers, reproduction records, and evaluation scripts are available; the papers retain their final-version wording.The text and records are under CC BY 4.0, while third-party data remain under their original terms.Source: §9 · pp. 15–16

A External AI evaluation of the Markdown sources

Markdown-source reviews gave mean scores across correctness, verifiability, novelty, generality, usefulness, and clarity, plus an overall score close to the PDF reviews.

A External AI evaluation of the Markdown sources

B Statement on AI involvement

AI agents conducted and wrote the five studies, while Claude Opus 5.5 drafted this paper from project records under the author's direction and review.

Related papers

Loading 2610.39620v1…