PaperlayerView exact revision on arXiv

Source-linked AI summary

NanoResearch: Co-Evolving Skills, Memory, and Policy for Personalized Research Automation

Jinhang Xu, Qiyuan Zhu, Yujun Wu, Zirui Wang, Dongxu Zhang, Marcia Tian, Yiling Duan, Siyuan Li, Jingxuan Wei, Sirui Han, Yike Guo, Odin Zhang, Conghui He, Cheng Tan

arXiv:2605.10813v2cs.AI

TL;DR

Research automation must accommodate researchers’ differing resources, preferences, and goals. NanoResearch co-evolves procedural skills, user-specific memory, and feedback-driven planning, outperforming existing systems while improving across successive research cycles.

  • Problem

    Research automation lacks reusable procedural knowledge, persistent user-specific experience, and implicit preference internalization despite researchers’ differing contexts and goals.

  • Method

    NanoResearch co-evolves a skill bank, user- and project-specific memory, and label-free feedback-driven policy updates to personalize multi-agent research planning.

  • Results

    Across 20 research topics spanning seven domains, NanoResearch consistently outperformed existing systems, producing higher-quality outputs, stronger preference alignment, and improving over successive cycles.

  • Takeaways & Limitations

    Personalized research automation requires complementary adaptation of procedural execution, contextual memory, and preference-guided planning rather than any single mechanism.

  • Takeaways & Limitations

    Evaluation centers on AI/ML tasks, leaving extension to disciplines requiring physical experimentation and instrument control as future work.

Abstract

from arXiv · show

LLM-powered multi-agent systems can now automate the full research pipeline from ideation to paper writing, but a fundamental question remains: automation for whom? Researchers operate under different resource configurations, hold different methodological preferences, and target different output formats. A system that produces uniform outputs regardless of these differences will systematically under-serve every individual user, making personalization a precondition for research automation to be genuinely usable. However, achieving it requires three capabilities that current systems lack: accumulating reusable procedural knowledge across projects, retaining user-specific experience across sessions, and internalizing implicit preferences that resist explicit formalization. We propose NanoResearch, a multi-agent framework that addresses these gaps through tri-level co-evolution. A skill bank distills recurring operations into compact procedural rules reusable across projects. A memory module maintains user- and project-specific experience that grounds planning decisions in each user's research history. A label-free policy learning converts free-form feedback into persistent parameter updates of the planner, reshaping subsequent coordination. These three layers co-evolve: reliable skills produce richer memory, richer memory informs better planning, and preference internalization continuously realigns the loop to each user. Extensive experiments demonstrate that NanoResearch delivers substantial gains over state-of-the-art AI research systems, and progressively refines itself to produce better research at lower cost over successive cycles.

1 Introduction

NanoResearch argues that personalized research automation is necessary because researchers differ in context and preferences, while existing systems apply one-size-fits-all pipelines. It addresses this gap through co-evolving skills, memory, and label-free policy learning, with experiments showing improving research quality and preference alignment.

  • Motivation: LLM-powered multi-agent systems now automate the full research lifecycle, including literature surveys, hypothesis generation, experiments, and paper writing.The introduction cites The AI Scientist, AI Scientist-v2, EvoScientist, and AI-Researcher as examples.
  • Motivation: Researchers differ in research values, resource budgets, methodological preferences, and target venues, making personalization a precondition for usable automation.Different communities prioritize real-world significance, architectural novelty, or benchmark gains.
  • Problem: Existing systems remain one-size-fits-all because they lack reusable procedural knowledge, cross-session user experience, and mechanisms for preserving implicit preferences.These gaps cause systems to repeatedly re-derive configurations, retain episodic rather than procedural memory, and lose nuanced feedback as context shifts.
  • Approach: NanoResearch addresses these gaps through tri-level co-evolution: a skill bank distills reusable rules, memory stores user- and project-specific records, and policy learning updates planner parameters from free-form feedback.The components are necessary together because procedural knowledge alone lacks user context, while contextual memory alone cannot prevent recurring failures.
  • Results: 20 research topics across seven domains show that NanoResearch outperforms existing systems in simulated and human evaluations while progressively improving output quality and preference alignment.The paper reports improvement over successive research cycles.

2 Related Work

Related work spans both end-to-end systems that automate the research lifecycle and task-specific tools that assist researchers at individual stages. End-to-end efforts have progressed from pioneering full pipelines toward more agentic exploration, while task-specific work builds on earlier AI-assisted research and foundation-model-based assistance.

  • End-to-end research automation: End-to-end research automation targets the full scientific lifecycle, from ideation to paper writing.The AI Scientist pioneered a fully automated pipeline culminating in LLM-based reviewing.
  • End-to-end research automation: AI Scientist-v2 adds agentic tree search to explore research decisions more extensively than its predecessor.This work extends the pioneering AI Scientist pipeline with additional agentic decision exploration.
  • Task-specific research automation: Task-specific research automation assists human researchers with individual subtasks rather than replacing them.This line of work predates the LLM era and continues through foundation-model-based assistance at specific research stages.
  • Task-specific research automation: Recent task-specific systems include literature-understanding tools such as PaperQA, which answers scientific questions.These efforts build on earlier AI support for scientific research and newer foundation-model assistance.

3 Method

NanoResearch coordinates a personalized, three-stage research pipeline through an Orchestrator that retrieves and updates reusable skills and project memories. Natural-language feedback is distilled into planner-parameter updates, enabling preferences to be internalized across successive cycles.

  • Framework overview: The Orchestrator converts a user-specified topic into a complete paper through ideation and planning, experimentation and optimization, and writing and review.A persistent user profile provides context, while the Skill Bank and Memory Module are retrieved before tasks and updated afterward.
  • Idea Generation and Planning: Ideation surveys literature, extracts quantitative evidence, proposes hypotheses, verifies novelty, and selects h* before generating a JSON-formatted experiment blueprint.The blueprint specifies datasets, baselines, architecture, metrics, and ablations, then undergoes iterative internal review until accepted or the retry limit is reached.
  • Experimental Validation and Optimization: Experimental validation stages prepare repositories and datasets, generate a self-contained codebase, debug execution autonomously, parse logs into analysis, and store outcomes and generalized solutions.Coding-specific skills and project memories guide the coding plan, while failed or successful experimental records remain available as project memory.
  • Paper Writing and Review: Writing integrates prior outputs into a publication-ready LaTeX manuscript drafted section-by-section, then applies independent review for coherence, claim validity, and formatting until quality thresholds are met.The review agent does not use the earlier skill or memory retrieval, and writing techniques and revision strategies are distilled afterward.
  • Skill and Memory Evolution: Context-aware retrieval prioritizes frequent, high-confidence skills and condition-matched project memories, while trajectory-based updating distills general rules into S and experiences into M.Semantically overlapping entries are merged to keep both stores compact for future cycles.
  • Preference Internalization: Free-form user feedback is encoded directly into the planner πθ through self-distillation, progressively internalizing fine-grained preferences across feedback rounds.This avoids relying on compression or retrieval through the Skill Bank or Memory Module for preferences such as coding style and analytical focus.

4 Experiments

NanoResearch is evaluated on personalized, multi-round research tasks spanning seven domains against four end-to-end systems using five lifecycle metrics. It surpasses baselines, improves across rounds, accumulates reusable knowledge, reduces cost, and transfers gains to real researchers.

  • Benchmark: The benchmark contains 20 research tasks across NLP, CV, Multimodal, Tabular ML, Time Series, Graph ML, and Audio, each paired with a preference- and constraint-specific simulated scientist.Claude generates topics and provides persona-consistent feedback after ideation, experimentation, and writing.
  • Evaluation setup: Systems are compared under identical task specifications using Compliance (Align.), Executability (E2E), Effectiveness (Perf.), Innovation (Novel.), and Expression metrics.Baselines are AI-Researcher, DeepScientist, EvoScientist, and AI Scientist-v2.
  • Baseline comparison: 100% end-to-end success in Round 1 makes NanoResearch the only system attaining perfect E2E, versus 50% for AI-Researcher and AI Scientist-v2 and 90% for DeepScientist.NanoResearch surpasses all baselines on every metric in Round 1.
  • Self-evolution: 8.963 vs. 6.656 on Compliance (Align.) is NanoResearch’s strongest Round 1 advantage, while Innovation rises from 4.960 to 5.645 and Expression from 5.428 to 6.172 by Round 3.Performance improves monotonically across all dimensions from Round 1 to Round 3.
  • Human evaluation: NanoResearch dominates four baselines on every dimension in Round 1 with three PhD researchers, while project performance increases from 0.6466 (R1) to 0.8603 (R3).This evaluation uses researchers’ own tasks and the same five-dimensional rubric.

5 Real Papers Produced by Our System

The system produced three complete papers end-to-end, demonstrating coherent manuscript generation across multiple research domains rather than isolated artifacts.

  • Three complete papers were produced end-to-end by the system and presented in Figures 6, 7, and 8.
  • The examples demonstrate that the pipeline can deliver coherent manuscripts, not only isolated ideas, code snippets, or tables.
  • The papers span sensor time-series classification, tabular regression, and audio keyword spotting.

6 Conclusion

The paper identifies personalization as a fundamental but overlooked dimension of research automation. It argues that accommodating real scientific practice requires simultaneous adaptation across procedural and contextual levels, rather than reliance on a single mechanism.

  • Personalization is fundamental to research automation because researchers differ in preferences, domain conventions, and evolving intent.A fixed pipeline cannot accommodate this diversity.
  • A single fixed pipeline cannot adequately support the diversity characterizing real scientific practice.
  • Effective personalization requires simultaneous adaptation at procedural and contextual levels rather than any single mechanism.The procedural level concerns how tasks are executed, while the contextual level concerns what prior experience is retained.

7 Limitations and Broader Impact … D.1 Shared Topic and Three Inductive Biases

NanoResearch is evaluated with explicit limitations, broader-impact considerations, and appendix judges for requirement alignment, novelty, and writing quality. A case study then holds the research topic and baselines fixed while showing how three user profiles produce different inductive biases, code interfaces, and writing tones.

  • 7 Limitations and Broader Impact: The evaluation focuses on AI/ML tasks realizable through code and text, while extending NanoResearch to experimentally grounded disciplines remains future work.Biology, chemistry, and physics are identified as non-trivial extensions because they often require physical experimentation and instrument control.
  • 7 Limitations and Broader Impact: NanoResearch lowers the barrier to automated research by adapting to preferences and reusing knowledge, but foundation-model dependence introduces computational and API costs.These costs may limit accessibility for resource-constrained researchers.
  • A User Requirement Alignment Prompt: The Compliance Score (Align.) measures whether generated artifacts satisfy stated task, dataset, baseline, ablation, compute-budget, and methodological-preference requirements.An LLM judge receives a structured JSON object containing the task specification, user requirements, generated idea or hypothesis, experiment plan or blueprint, and available benchmark or execution status.
  • B Novelty Evaluation Prompt: The Novelty Score uses a 1–10 LLM-judged rubric that prioritizes core mechanisms and penalizes superficial changes such as backbone swaps, tuning, regularization, or data augmentation.The judge compares the proposed idea with provided baselines and returns novelty_score, closest_baseline, and rationale in JSON.
  • C Writing Quality Evaluation Prompt: Overall Writing Quality and its Fluency, Motivation Clarity, and Preference Alignment sub-dimensions are judged from the full draft on a strict 1–10 scale.The rubric spans very poor writing through excellent, submission-quality writing and considers readability, organization, motivation clarity, scientific tone, and consistency with requested style.
  • D Case Study: Three Profiles, One Topic: The case study adapts one research topic to three user profiles by varying the proposed method’s inductive bias, code interface, and writing tone while holding the topic and baselines fixed.Intermediate outputs are shown for the experiment blueprint, coding output, and writing style across pipeline Stages I–III.
  • D.1 Shared Topic and Three Inductive Biases: The shared case-study task is lightweight time-series sensor classification on UCI HAR, using comparisons with a 1D CNN, a GRU, and InceptionTime-small under a single-GPU feasibility constraint.The three profiles therefore impose different inductive biases over the same UCI HAR input, summarized in Table 7.

D.2 Profile A: Evidence-First Scientist

Profile A proposes FixedMultiScaleResidualCNN, a compact 1D CNN that tests fixed temporal receptive fields at scales 3, 5, and 7 for controlled UCI HAR classification. Its evaluation isolates kernel-scale, residual, and parameter-count effects under matched conditions.

  • Experiment blueprint: FixedMultiScaleResidualCNN augments a standard 1D CNN with parallel fixed-kernel temporal branches at scales 3, 5, and 7, followed by feature fusion and residual projection.The design targets broader fixed receptive fields while remaining close to the baseline.
  • Experiment blueprint: The architecture processes 9-channel, 128-step inputs through parallel Conv1d branches, concatenation, 1×1 projection, residual addition, pooling, and linear classification.Its key components are fixed multi-scale Conv1d branches, residual feature fusion, and a compact classification head.
  • Experiment blueprint: The ablations compare kernel_sizes=[5] with [3] [5] [7], remove the skip connection, and match channel width to control parameter count.These tests separate the effects of kernel scale, residual connections, and model capacity.
  • Writing style: The study asks whether fixed multi-scale temporal features yield reproducible gains under matched training conditions without sample-adaptive routing.The encoder changes only the temporal receptive-field structure of a standard 1D CNN.

D.3 Profile B: Ablation-Focused Researcher

Profile B centers on a lightweight TemporalFeatureGating module whose contribution can be isolated from a standard 1D CNN backbone. Its central claim is tested through removal, static-gate, and bottleneck-size ablations to determine whether learned temporal weighting explains the accuracy–cost tradeoff.

  • Experiment blueprint: TemporalFeatureGating inserts a lightweight temporal gate after the 1D CNN extractor to weight time steps before global pooling.The module is separated from the backbone, data loader, optimizer, and classifier so its contribution can be isolated.
  • Experiment blueprint: The architecture processes 9 channels × 128 time steps through a 1D CNN, temporal-gate MLP, feature reweighting, global average pooling, and linear classification.Key components include a small gate bottleneck, weighted temporal pooling, and a compact classification head.
  • Ablation design: Ablations compare gate removal, uniform static temporal weights, and full versus tiny bottlenecks while leaving the backbone unchanged.The gate can be removed, frozen to uniform weights, or compressed to test whether learned temporal weighting drives the observed behavior.
  • Writing style: The central claim is that a removable temporal gate improves a 1D CNN only if its learned weighting survives direct removal, static-gate, and tiny-gate ablations.This framing directly tests whether learned temporal weighting explains the observed accuracy–cost tradeoff.

D.4 Profile C: Benchmark-Driven Exploratory Researcher

Profile C develops TemporalEvidenceRouting, a benchmark-facing sensor-classification model that dynamically routes each sample among short-, medium-, and long-range temporal experts. Its design supports benchmark comparisons against fixed CNN, recurrent, and Inception-style compact baselines.

  • Stage I: Experiment blueprint: TemporalEvidenceRouting uses a lightweight sample-adaptive router to select among short-, medium-, and long-range convolutional experts for multi-scale temporal feature extraction.The model includes a shared classification head and supports broader benchmark-facing comparisons.
  • Stage I: Experiment blueprint: The architecture processes 9 channels × 128 time steps with kernel sizes [3] [5] [9], then performs weighted expert fusion, global pooling, and linear classification.The router predicts expert weights per sample before fusion.
  • Stage I: Experiment blueprint: The experiments compare single-expert, uniform-routing, and no-router feature-fusion ablations while analyzing routing statistics across activity classes.These ablations isolate the contribution of adaptive routing and expert diversity.
  • Stage III: Writing style: The writing frames lightweight sensor classification as temporal evidence routing, where each input dynamically selects the expert needed for strong benchmark performance against fixed CNN, recurrent, and Inception-style compact baselines.This framing presents multi-scale temporal modeling as a sample-adaptive decision.

E Example of User Profile, Memory, and Skill

The examples show how NanoResearch turns a researcher’s preferences into an explicit profile, project-specific memories, and reusable experiment-design skills. For an evidence-first scientist, these components prioritize reproducibility, controlled comparisons, conservative methods, and direct ablations.

  • User Profile: Profile A prioritizes exact reruns, explicit controls, reproducible ablations, conservative standard PyTorch methods, and restrained evidence-grounded writing.The profile favors reproducibility and controlled experimentation over speculative novelty.
  • Memory: The profile’s UCI HAR memory requires reproducing a compact 1D CNN, GRU, and InceptionTime-small baseline suite before evaluating a new method.All models should use the same split, preprocessing, optimizer, learning rate, batch size, epoch budget, and evaluation script.
  • Memory: A second memory fixes data processing, optimization, training, stopping, seed, and metric controls across baselines, proposed methods, and ablations.Unavoidable implementation differences must be explicitly logged and justified.
  • Memory: A low-risk architectural memory recommends adding a small temporal module to a recognizable 1D CNN backbone rather than replacing the full architecture.The change should test a narrow temporal-feature hypothesis while preserving baseline training behavior and enabling attribution to one modeling change.
  • Skill: The reusable skills require an explicit deterministic experiment contract and mapping every method claim to a direct ablation.The deterministic contract includes fixed seeds, saved split indices, software and configuration logging, and shared evaluation scripts.

Profile C: Benchmark-Driven Exploratory Researcher

Profile C describes a high-novelty exploratory researcher who prioritizes broad benchmark coverage, clear leaderboard-facing evidence, and named mechanisms with visible empirical upside. The profile accepts higher-risk proposals when novelty and compute constraints are explicit, within a 2×A100 80GB, 5-day budget.

  • The researcher favors strong benchmark coverage, broad comparisons, clear leaderboard-facing evidence, and a named mechanism with visible empirical upside.
  • Higher-risk proposals are acceptable when novelty and compute constraints are explicit, with a resource budget of 2×A100 80GB, 5 days.
  • The profile rejects novelty claims unsupported by experiment or literature positioning and weak benchmark coverage or insufficient comparison breadth.
  • Benchmark-facing claims should use comparison breadth, resource metrics, and failure or sensitivity analysis rather than relying on a single best-number improvement.
  • The reusable skills require converting local architectural changes into observable concept-level mechanisms and pairing strong claims with benchmark, resource, and failure evidence.

F Research Topics Specification

The evaluation uses 20 research topics spanning seven domains, with diverse task types, modalities, and evaluation setups. The listed topics emphasize practical, reproducible improvements under constrained compute across NLP, vision, multimodal, tabular, time-series, and graph learning.

  • Scope: 20 research topics span seven domains and cover diverse task types, modalities, and evaluation setups.The complete topic list is provided in Listing 1.
  • NLP: NLP topics target lightweight improvements for biomedical question answering, short-text classification, and sentence-pair classification or matching.They prioritize reproducibility, compact methods, clean ablations, and avoidance of heavy retrieval or multi-stage pipelines, using PubMedQA, AG News, SST-2, MRPC, and RTE.
  • Vision, multimodal, and tabular ML: Computer-vision, multimodal, and tabular topics require benchmarkable methods that improve quality without oversized models, expensive pretraining, large ensembles, or impractical stacks.The associated settings include CIFAR-10 and FashionMNIST with ResNet-18, MobileNetV3-small, and ViT-tiny; MMMU and ScienceQA with compact VLM and late-fusion baselines; and Adult and CoverType with XGBoost, TabTransformer, and MLP baselines.
  • Time Series and Graph ML: Time-series and graph topics test compact modeling refinements on sensor and citation-graph benchmarks under modest infrastructure requirements.The settings use UCI HAR with 1D CNN, GRU, and InceptionTime-small baselines, and Cora and Citeseer with GCN, GraphSAGE, and GAT baselines.
Loading 2605.10813v2…