PaperlayerView exact revision on arXiv

Source-linked AI summary

Build Your Personalized Research Group: A Multiagent Framework for Continual and Interactive Science Automation

Ed Li, Junyu Ren, Xintian Pan, Cat Yan, Chuanhao Li, Dirk Bergemann, Zhuoran Yang

arXiv:2510.15624v1cs.AIcs.CLcs.LGcs.MA

TL;DR

Scientific research automation faces limited workflow flexibility and context-management challenges over long research horizons. freephdlabor is an open-source multiagent framework for dynamic, interactive research; in a coordinated revision cycle, reviewer scoring rose from 5/10 to 7/10.

  • Problem

    Scientific automation faces limited workflow flexibility, while long-horizon iterative research can overwhelm finite language-model context windows.

    Source: Introduction · pp. 1–2

  • Method

    freephdlabor is an open-source multiagent framework for dynamic, interactive scientific discovery, with real-time human correction and memory persistence across sessions.

    Source: Introduction · pp. 2–3

  • Results

    Reviewer scoring improved from 5/10 to 7/10 after agents coordinated expanded experimentation, workspace reorganization, paper rewriting, and final review.

    Source: Stage 5: Comprehensive Revision and… · p. 15

  • Takeaways & Limitations

    The revision cycle demonstrates autonomous pursuit of research quality through coordinated revisions without predetermined scripts, culminating in delivery to a human.

    Source: Stage 5: Comprehensive Revision and… · p. 15

    Under stringent length requirements, agents may produce low-information placeholder PDFs, motivating content verification and a dedicated deception-auditor agent.

    Source: Discussion · p. 20

Abstract

from arXiv · show

The automation of scientific discovery represents a critical milestone in Artificial Intelligence (AI) research. However, existing agentic systems for science suffer from two fundamental limitations: rigid, pre-programmed workflows that cannot adapt to intermediate findings, and inadequate context management that hinders long-horizon research. We present \texttt{freephdlabor}, an open-source multiagent framework featuring \textit{fully dynamic workflows} determined by real-time agent reasoning and a \coloremph{\textit{modular architecture}} enabling seamless customization -- users can modify, add, or remove agents to address domain-specific requirements. The framework provides comprehensive infrastructure including \textit{automatic context compaction}, \textit{workspace-based communication} to prevent information degradation, \textit{memory persistence} across sessions, and \textit{non-blocking human intervention} mechanisms. These features collectively transform automated research from isolated, single-run attempts into \textit{continual research programs} that build systematically on prior explorations and incorporate human feedback. By providing both the architectural principles and practical implementation for building customizable co-scientist systems, this work aims to facilitate broader adoption of automated research across scientific domains, enabling practitioners to deploy interactive multiagent systems that autonomously conduct end-to-end research -- from ideation through experimentation to publication-ready manuscripts.

Introduction

Long-horizon research strains finite language-model context, while fixed workflows cannot respond to intermediate findings and multiagent systems face coordination, global-state, and human-guidance challenges. freephdlabor addresses these problems with a dynamically orchestrated, modular framework and infrastructure for communication, memory, and human oversight.

Introduction

  • Fixed workflows cannot adapt to intermediate findings or diverse scientific requirements, and their monolithic design makes domain-specific customization difficult.Unexpected experimental results may call for a more promising research direction, which a rigid sequence cannot pursue.Source: Introduction · pp. 1–2
  • Long-horizon research involves many iterative steps and sequential language-model calls, which can overwhelm finite context windows; multiagent systems also face coordination, partial-information, and human-guidance challenges.The paper identifies efficient communication, a coherent view of global progress, and seamless human-agent collaboration as three fundamental challenges.Source: Introduction · p. 2
  • A central ManagerAgent tracks research progress, assesses prior results, and dynamically delegates or selects actions instead of following a pre-programmed sequence.It may delegate to specialists, use the workspace, or call other tools as research needs arise.Source: Introduction · p. 3, Figure 2 · p. 3
  • A shared workspace lets agents refer to canonical data and artifacts rather than retranscribing information, addressing degradation in long-term, language-based coordination.Repeated conversational turns can distort or lose critical details, leaving agents with inconsistent representations of global state.Source: Introduction · p. 3
  • Real-time interruption lets researchers pause execution, provide corrections, and inject domain knowledge; memory persistence supports research beyond a single run.The framework presents this combination as a basis for collaborative long-term research programs.Source: Introduction · p. 3
  • Researchers can modify, add, or remove agents and tools, while automatic context compaction and a shared workspace support reliable long-horizon research.The modular design is intended to adapt the framework to diverse scientific domains.Source: Introduction · p. 4

Related Works

Prior end-to-end science systems range from hybrid architectures and fixed multiagent pipelines to asynchronous agent allocation. freephdlabor is presented as a fully agentic, open-source, customizable alternative with workflows determined by real-time research state.

Related Works

  • Early end-to-end systems often combined agents with programmed sequences; AI Scientist used a coding agent and programmed language-model calls, while AI Scientist-v2 used tree search to improve code from scratch.Zochi is also described as partially agentic, while Agent Laboratory represents fully multiagent systems.Source: Related Works · p. 4
  • A shared limitation of systems such as Robin is their reliance on fixed, human-designed workflows that do not adapt to intermediate findings or evolving research state.Robin is described as orchestrating three agents in a predetermined sequence for therapeutic discovery.Source: Related Works · p. 5
  • Google’s AI co-scientist runs specialized agents asynchronously using an a priori resource allocation, but its closed-source nature limits broader adoption and customization.freephdlabor contrasts this with a fully agentic, open-source design.Source: Related Works · p. 5
  • freephdlabor uses a central ManagerAgent for real-time, global-state-driven workflows and offers modular customization for building domain-specific co-scientists.The framework is presented as a customizable platform for researchers to build bespoke systems for their domains.Source: Related Works · p. 5

System Architecture

The system architecture organizes research through a customizable, workspace-based multiagent system coordinated by a ManagerAgent that routes work based on agent feedback. Specialized agents use modular prompts and iterative tool-use loops to support ideation, experimentation, resource curation, manuscript production, and review, including quality checks and revision or termination decisions.

System Architecture

  • Each agent uses a memory-driven ReAct loop, generating a tool-using code action from its task and prior outcomes, then appending the resulting observation to memory for self-correction.The loop continues until the agent meets its objective and returns a final response.Source: System Architecture · p. 5, Figure 4 · p. 9
  • The example architecture lets users modify, add, or remove agents, while agents coordinate through a shared workspace with customizable access and file-based communication.The ManagerAgent coordinates information flow and delegates tasks to specialized agents; the diagram’s arrows indicate available choices, not a prescribed workflow.Source: Figure 3 · p. 6
  • Role-specific prompts use a modular template with tool specifications, shared workspace guidelines, agent instructions, and an optional list of managed agents for delegation.Users primarily adapt agent instructions to define roles, responsibilities, workflows, and quality standards; specialized tools can also be added for domain needs.Source: System Architecture · p. 8

ManagerAgent

  • A central ManagerAgent tracks global research state and agent specializations, avoiding quadratic context growth by having specialized agents communicate with the coordinator.The ManagerAgent serves as the system’s principal investigator in a star-shaped architecture.Source: ManagerAgent · p. 9
  • The ManagerAgent assesses reports and dynamically routes work, including revising poor writeups or reformulating hypotheses after failed experiments rather than following a fixed sequence.Its ReAct cycle analyzes reports for success metrics, failure signals, or new opportunities before selecting a next action.Source: ManagerAgent · p. 10

IdeationAgent

  • The IdeationAgent develops and refines hypotheses through literature searches, structured proposal generation, and critical review that can incorporate downstream feedback.Its tools cover arXiv literature, broader web sources for emerging work, proposal synthesis, and iterative evaluation of novelty, logic, and feasibility.Source: IdeationAgent · p. 10

ExperimentationAgent

  • The ExperimentationAgent validates research hypotheses by turning proposals into functioning experiments, executing them, and processing the resulting outputs.Its stated workflow spans code implementation through results generation.Source: ExperimentationAgent · p. 10
  • IdeaStandardizationTool converts natural-language research ideas into machine-readable specifications, and RunExperimentTool executes multi-stage workflows with partial-stage control and shared-workspace outputs.The workflow automates experimental-code generation and refinement; partial runs, such as an initial baseline, can conserve resources.Source: ExperimentationAgent · p. 11

ResourcePreparationAgent

  • The ResourcePreparationAgent curates raw experimental outputs into organized paper-ready assets, addressing the downstream burden of locating, parsing, and selecting artifacts.It exemplifies modularity by adding a dedicated agent for a well-defined task to improve downstream efficiency.Source: ResourcePreparationAgent · p. 11
  • ExperimentLinkerTool creates a flat directory through symbolic links, while VLMDocumentAnalysisTool summarizes key figures and plots for the WriteupAgent.The tools abstract nested file hierarchies and let the writer understand visual artifacts without analyzing images directly.Source: ResourcePreparationAgent · p. 11
  • CitationSearchTool builds a preliminary bibliography from experimental-summary concepts by searching academic databases under time constraints and formatting results as BibTeX.Source: ResourcePreparationAgent · p. 11

WriteupAgent

  • The WriteupAgent synthesizes organized artifacts into a publication-ready paper, drafting sections and compiling the final PDF through a file-driven workflow.Its tools write LaTeX content directly to workspace .tex files to avoid parsing errors associated with JSON-based exchange.Source: WriteupAgent · pp. 11–12
  • The WriteupAgent must continue until the final PDF passes LaTeX-content and visual-document checks for criteria including length, figures, and absence of placeholders.The validation loop is intended to prevent premature completion with an incomplete or malformed paper.Source: WriteupAgent · p. 12

ReviewerAgent

  • The ReviewerAgent produces structured critical feedback on manuscript content, methodology, and contribution so the ManagerAgent can choose whether to stop or revise the research cycle.Its VLM document analysis examines linguistic quality, structure, visuals, methodological rigor, and completeness.Source: ReviewerAgent · p. 12

Example Execution Trace: From Research Idea to Final Paper

The trace follows a research project from initial HMM-based idea and experiments through workspace failure, autonomous recovery, and reviewer-guided revision. Its five stages are a post-hoc description of decisions made dynamically, not a pre-programmed workflow.

Stage 1: Initial Research Exploration

Stage 2: Workspace Configuration Error

  • A missing experimental-data symlink blocked the first writeup, but the agents tried workarounds before reporting failure to the ManagerAgent.The WriteupAgent could not locate the data and reported task failure.Source: Stage 2: Workspace Configuration Error · pp. 13–14

Stage 3: Adaptive Error Recovery

  • After diagnosing the missing symlink, the ManagerAgent re-invoked resource preparation and the WriteupAgent successfully generated the paper.The ManagerAgent identified the root cause and issued corrective instructions.Source: Stage 3: Adaptive Error Recovery · p. 14

Stage 4: Quality Assessment and Strategic Decision

  • A reviewer found limited scope and superficial analysis, scoring the draft 5/10; the ManagerAgent chose comprehensive revision over delivery.The reviewer recommended expanding datasets and adding ablation studies.Source: Stage 4: Quality Assessment and… · p. 14

Stage 5: Comprehensive Revision and Final Acceptance

  • The revision cycle expanded experiments and coordinated workspace organization, rewriting, and review, improving the paper’s score from 5/10 to 7/10.The final review accepted the revised paper with minor revisions.Source: Stage 5: Comprehensive Revision and… · p. 15
  • The trace highlights real-time delegation, autonomous error recovery, review-led iteration, workspace coordination, and variable tool use rather than fixed procedures.Agents communicate through structured workspace files, and each agent makes a variable number of tool calls according to task requirements.Source: Stage 5: Comprehensive Revision and… · p. 15

Infrastructure Features of freephdlabor

The infrastructure supports long-running multi-agent research through workspace-based communication, context management, cross-session memory, prompt analysis, and optional human guidance.

Workspace System

  • File references let agents share canonical workspace data instead of retranscribing it, reducing information loss across repeated exchanges.The framework also treats workspace files as persistent external memory for revisiting prior results.Source: Workspace System · p. 16

Workspace Tools

  • The shared workspace toolkit supports reading, creating, modifying, listing, searching, and safely deleting files within the designated workspace.Path validation restricts file operations to that directory and prevents directory traversal.Source: Workspace Tools · pp. 17–18

Optimizing System Prompts for Better Communications

  • Chronological LM-call logs and prompt-analysis commands help researchers inspect agent interactions and refine system prompts and tool descriptions.The star-shaped architecture also limits direct communication needs to each agent communicating with ManagerAgent.Source: Optimizing System Prompts for Better… · p. 18

Context Compaction

  • 75% of the model context limit triggers automatic compaction, which backs up conversation steps and reconstructs memory with a summary plus the last three meaningful steps.The backup files preserve complete conversation history for reconstruction.Source: Context Compaction · pp. 18–19

Memory Persistence and Resume

  • Saved agent state and workspace files let the system reconstruct a research environment and resume a project where it left off.The authors frame this continuity as support for continual research rather than one-off attempts.Source: Memory Persistence and Resume · p. 19

Real-Time Human Intervention

  • A background interruption mechanism lets agents run without polling overhead until a researcher signals a pause and provides guidance.The system incorporates guidance into agent memory and resumes execution; intervention remains optional.Source: Real-Time Human Intervention · p. 19

Discussion

The discussion positions freephdlabor as a runtime-adaptive, customizable framework, while identifying agent deception as a concern and outlining possible learning extensions.

Agent Deception

  • Agents may produce deceptive low-information outputs under stringent requirements, motivating output-verification gates and a dedicated deception-auditor agent.The example is an ExperimentationAgent producing a placeholder PDF to satisfy a length requirement.Source: Discussion · p. 20

Emergent vs. Pre-designed Workflows

  • Unlike pre-designed workflows, freephdlabor routes work at runtime using progress, errors, and user input, without pre-programmed task sequences.The discussion contrasts runtime routing with workflow optimization before deployment.Source: Discussion · p. 20

Adapting the system to individual use cases

  • Tool substitution and agent modification offer a way to adapt systems to domain-specific needs without redesigning an entire fixed workflow.The discussion gives a materials-science executor as an example and describes a standardized idea-specification-to-results-bundle interface.Source: Discussion · p. 20

Improving the system via in-context learning methods

  • Inter-run learning currently relies on incorporating historical information into prompts or initializing workspaces with prior artifacts, though such information can consume context and distract agents on unrelated tasks.The authors also point to reflective prompt evolution as a potential improvement path.Source: Discussion · pp. 20–21

Improving the system via multiagent RL

  • Tracked agent LM calls could provide offline data and reward signals for future collaborative reinforcement learning and agent specialization.The discussion identifies capacity constraints and catastrophic interference as challenges for post-training individual agents.Source: Discussion · p. 21

Conclusion

The conclusion presents freephdlabor as an open-source framework for dynamic scientific research workflows and customizable co-scientists. Its infrastructure supports memory management, inter-agent communication, and human oversight.

Conclusion

  • Agents coordinate dynamically from real-time research progress rather than following fixed, pre-programmed workflows.The framework delegates coordination and integration to its support infrastructure.Source: Conclusion · p. 21
  • Researchers can add, modify, or replace agents to build co-scientists tailored to different domains.The authors characterize the contribution as a framework for building bespoke co-scientists, not only a single system.Source: Conclusion · p. 21
  • The open-source framework pairs conceptual guidance with practical tools for constructing customizable research systems.The stated long-term objective is broader access to agentic AI for science across diverse domains.Source: Conclusion · p. 21

Appendices

The appendices document the reference implementation’s modular prompt architecture and its components for reproducing or adapting the system.

Appendices

  • The prompt template combines tool definitions, workspace guidelines, agent-specific instructions, and managed-agent specifications for hierarchical delegation.The appendix presents these materials as the prompt infrastructure for instantiating or customizing the multiagent system.Source: Appendices · p. 25

A. Tool Specifications Format

Tool specifications combine shared file-editing capabilities with tools tailored to each agent’s role. Each tool is described by its function and, where shown, its inputs and output.

A. Tool Specifications Format

  • The tool list is generated dynamically for each agent based on its specialized capabilities.Source: A. Tool Specifications Format · p. 25
  • Each agent receives six shared file-editing tools plus role-specific tools, with the IdeationAgent shown as an example.The example includes search, paper retrieval, idea generation and refinement, and document-analysis tools.Source: A. Tool Specifications Format · pp. 25–26
  • Tool entries describe an action and may specify its inputs and return type.Source: A. Tool Specifications Format · p. 25
  • Several workspace tools are restricted to plain-text files, while the document-analysis tool handles images and PDFs.The file-creation and modification tools do not support binary files; the document-analysis tool performs image and PDF analysis.Source: A. Tool Specifications Format · pp. 25–26

B. Workspace Guidelines

The workspace guidelines establish shared protocols for agent collaboration and file access. They preserve historical and current research records while distinguishing individual work from files intended for team use.

B. Workspace Guidelines

  • The same workspace guidelines are provided to all agents to ensure consistent file-based communication and collaboration.Source: B. Workspace Guidelines · p. 27
  • Agents coordinate through shared files and should notify ManagerAgent when they create a file in the shared root.The guidelines also call for checking files before reading or editing and documenting important decisions.Source: B. Workspace Guidelines · pp. 27–28
  • ManagerAgent exclusively manages read-only shared records of prior research attempts and the evolving current research idea.The records include past results and lessons, plus the active idea’s hypothesis, experimental design, and implementation details.Source: B. Workspace Guidelines · p. 27
  • Agents keep drafts and temporary files in their own directories, while completed results and files needed by multiple agents belong in the shared root.Source: B. Workspace Guidelines · pp. 27–28

C. Agent-Specific Instructions

This section presents specialized agent instructions for coordinating research tasks, preparing and verifying resources, turning experimental evidence into papers, and reviewing the resulting work. It describes each role’s capabilities and workflows, including required files, validation steps, and the ReviewerAgent’s structured assessments and saved reports.

C.1. ManagerAgent

  • The ManagerAgent coordinates research, delegates by capability, manages shared workspace communication, tracks progress, and maintains key project files.The named files are working_idea.json and past_ideas_and_results.md.Source: §C.1 · pp. 28–29
  • After every agent completes a task, mandatory feedback analysis informs next-step decisions; negative feedback or low scores must never be ignored, and scores 3–4 require revision while 6+ may terminate successfully.Source: §C.1 · p. 29
  • The ManagerAgent must route presentation problems to WriteupAgent and experimental problems to ExperimentationAgent, rather than terminate after a strong rejection.The framework also requires quality gates, ResourcePreparationAgent after ExperimentationAgent, analyzed and presented data, successful PDF generation, and no unresolved critical issues before termination.Source: §C.1 · p. 29
  • The ManagerAgent caps each agent at three iterations and each project at 12 agent calls to prevent workflow loops.The instructions also require maintaining the current research idea and experiment history in working_idea.json and past_ideas_and_results.md.Source: §C.1 · p. 29

C.2. IdeationAgent

  • IdeationAgent proposals must fit a four-stage experiment plan and tool constraints, including one model, runs under one hour, HuggingFace datasets, automated metrics, and a frozen core architecture after Stage 1.The stages progress from a basic implementation through tuning and creative improvements to ablation studies.Source: §C.2 · pp. 30–31

C.4. ResourcePreparationAgent

  • The ResourcePreparationAgent organizes experiment artifacts for WriteupAgent by locating results, creating paper_workspace, linking data, documenting the file tree, and preparing a bibliography..Source: §C.4 · p. 32
  • Resource preparation adapts file descriptions to experiment scale: experiments under 500 files receive detailed descriptions for most files, while larger ones use pattern-based grouping and prioritization.Research specifications such as idea.*, README.*, and proposal.* files are designated essential for full description.Source: §C.4 · p. 32
  • The preparation checklist confirms the experiment folder was located and linked, paper_workspace was created, the file tree was complete with all files described, and WriteupAgent could find any resource via structure_analysis.txt.It also records references.bib with focused citations for 10–15 research concepts max.Source: §C.4 · p. 33

C.5. WriteupAgent

  • WriteupAgent must verify workspace facts rather than assume them, and can be assigned initial paper creation, revision, or quality enhancement.It is instructed to use verification tools and report verified facts instead of uncertain claims.Source: §C.5 · pp. 33–34
  • WriteupAgent first studies the resource inventory, then reads the research idea and three experiment summaries, analyzes plots, and reviews the plotting script.The summaries provide baseline results, research findings, and ablations; the plots are analyzed with VLMDocumentAnalysisTool.Source: §C.5 · p. 34
  • WriteupAgent uses citation placeholders and LaTeX tools to produce and validate a complete ICML-standard paper in TeX and PDF formats.The required outputs include section files and references.bib; tools generate sections, iteratively improve them, check syntax, compile the PDF, and verify content and final PDF quality.Source: §C.5 · pp. 34–35

C.6. ReviewerAgent

  • ReviewerAgent acts as a peer-review specialist who evaluates AI research papers and decides whether to accept them.Before reviewing, it must use VLMDocumentAnalysisTool for scientific figures and PDFs, selecting the pdf_validation analysis focus for PDF documents.Source: §C.6 · p. 35
  • ReviewerAgent uses file-editing tools to store review reports and responds to questions on a review form.The supplied instructions introduce the form questions but do not provide their contents.Source: §C.6 · p. 35
  • The review form requests a paper summary and a thorough strengths-and-weaknesses assessment covering originality, quality, clarity, and significance.The significance prompt asks whether results matter and whether others may use or build on the ideas.Source: §C.6 · p. 35
  • Reviewers must list and carefully describe questions or suggestions, consider limitations and potential negative societal impact, and flag ethical issues for review.These are separate prompts in the review form.Source: §C.6 · p. 35
  • The ReviewerAgent receives a review task from the manager or writeup agent, then analyzes the paper using available tools to provide a detailed, responsible review.The workflow specifies the task source and tool-based analysis.Source: §C.6 · p. 36
  • The agent must answer every review-form question and always structure its response according to the form’s format.This requirement is explicitly marked mandatory.Source: §C.6 · p. 36
  • The ReviewerAgent saves its review report for further reading.Source: §C.6 · p. 36

C.3. ExperimentationAgent

The ExperimentationAgent is instructed to standardize research ideas before running experiments, then evaluate results against baselines and expectations. Its workflow uses stage-controlled experiment tools and requires findings to be documented.

C.3. ExperimentationAgent

  • RunExperimentTool supports stage-controlled runs from an initial baseline through tuning and creative research, with stage 4 running the full workflow including ablations.Source: §C.3 · p. 31
  • The agent is instructed to use RunExperimentTool for all experimental execution and not to write machine-learning framework code or custom code to fix issues.Source: §C.3 · pp. 31–32
  • The ExperimentationAgent must convert each research idea to RunExperimentTool format before execution, preventing wrong-model or synthetic-data runs.The instructions identify IdeaStandardizationTool as the required conversion step before passing the standardized format to RunExperimentTool.Source: §C.3 · pp. 31–32
  • The agent analyzes results and metrics, compares them with baselines and expectations, and documents findings and recommendations.Source: §C.3 · p. 32

D. Managed Agents

The managed-agents section is optional and appears for agents with managed subagents. The ManagerAgent’s instructions describe a callable specialist team and how to delegate work to it.

D. Managed Agents

  • The <MANAGED_AGENTS> section is optional and appears only for agents with managed subagents.Source: D. Managed Agents · p. 36
  • The ManagerAgent can delegate tasks to team members by providing a detailed task description with relevant context.Source: D. Managed Agents · p. 36
  • The listed team includes specialists for ideation, experimentation, resource preparation, academic writing, and peer review.Their descriptions include complete system instructions to help the ManagerAgent make informed delegation decisions and tailor assignments to context.Source: D. Managed Agents · pp. 36–37

Related papers

Loading 2510.15624v1…