PaperlayerView exact revision on arXiv

Source-linked AI summary

ResearchTown: Simulator of Human Research Community

Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, Jiaxuan You

arXiv:2412.17767v2cs.CLcs.LG

TL;DR

The paper asks whether LLMs can simulate human research communities in a way that supports understanding and discovery of research ideas. RESEARCHTOWN represents researchers and papers as an agent-data graph and models reading, writing, and reviewing with TextGNN, evaluated through ResearchBench. It reports realistic and robust collaborative simulation and generates interdisciplinary ideas, while noting reduced quality and coherence in some out-of-distribution combinations.

  • Problem

    Whether LLMs can simulate human research communities remains open, despite their potential as scientific-research copilots.

  • Method

    RESEARCHTOWN uses an agent-data graph with LLM-based TextGNN message passing to simulate research activities and evaluates outputs with similarity-based node masking.

  • Results

    RESEARCHTOWN realistically simulates collaborative paper and review writing, remains robust with multiple researchers and diverse papers, and generates interdisciplinary research ideas.

  • Takeaways & Limitations

    RESEARCHTOWN provides a platform for studying research dynamics and supporting automated scientific discovery through agent collaboration.

  • Takeaways & Limitations

    When combining too many disparate domains, RESEARCHTOWN can produce incoherent or superficial outputs that limit real-world use.

Abstract

from arXiv · show

Large Language Models (LLMs) have demonstrated remarkable potential in scientific domains, yet a fundamental question remains unanswered: Can we simulate human research communities with LLMs? Addressing this question can deepen our understanding of the processes behind idea brainstorming and inspire the automatic discovery of novel scientific insights. In this work, we propose ResearchTown, a multi-agent framework for research community simulation. Within this framework, the human research community is simplified as an agent-data graph, where researchers and papers are represented as agent-type and data-type nodes, respectively, and connected based on their collaboration relationships. We also introduce TextGNN, a text-based inference framework that models various research activities (e.g., paper reading, paper writing, and review writing) as special forms of a unified message-passing process on the agent-data graph. To evaluate the quality of the research community simulation, we present ResearchBench, a benchmark that uses a node-masking prediction task for scalable and objective assessment based on similarity. Our experiments reveal three key findings: (1) ResearchTown can provide a realistic simulation of collaborative research activities, including paper writing and review writing; (2) ResearchTown can maintain robust simulation with multiple researchers and diverse papers; (3) ResearchTown can generate interdisciplinary research ideas that potentially inspire pioneering research directions.

1. Introduction

RESEARCHTOWN addresses whether LLMs can simulate human research communities by representing researchers and research materials in an agent-data graph and applying TextGNN inference. It introduces ResearchBench for scalable similarity-based evaluation and reports realistic, robust, and interdisciplinary simulation outcomes.

  • Research community as graph: RESEARCHTOWN models researchers as agent nodes and papers, reviews, and blogs as data nodes, with agent nodes functioning over data nodes.The framework can also represent other research content and interactions, including codebases, model releases, and social-media posts.
  • Novel framework for research simulation: TextGNN extends graph message passing to text-based inference for simulating collaborative research activities such as paper reading, writing, and reviewing.The framework uses LLM-based agent functions to process text information over the agent-data graph.
  • Novel evaluation for research simulation: ResearchBench evaluates simulation similarity by masking paper nodes and testing whether the simulator reconstructs them, avoiding the subjectivity and cost of human evaluation.The benchmark contains 1,000 paper-writing tasks and 200 review-writing tasks requiring multi-agent collaboration.
  • Main discoveries: RESEARCHTOWN achieves average similarity scores of 0.68 for paper writing and 0.49 for review writing, while improving with more agents and retaining performance with unrelated papers.These findings support realistic simulation of collaborative research activities and robustness across researchers and papers.
  • Main discoveries: RESEARCHTOWN generates interdisciplinary ideas combining NLP, criminology, and astronomy that do not exist in the real-world research data.The paper presents this as a potential source of new research directions.
  • Stressing ethical concerns: The framework raises ethical concerns about facilitating plagiarism and producing low-quality or misleading claims.The paper states that these concerns are addressed in Appendix §B.

4. Building TextGNN on Agent-Data Graphs

TextGNN adapts GNN-style message passing to textual hidden states on an agent-data graph. Its layers use LLM agent functions to generate and aggregate text messages across agent-agent, agent-data, and data-data interactions.

  • Message passing in TextGNN: TextGNN replaces embedding-space hidden states with text-space states, enabling LLMs to perform message passing over agent-data graphs.Data nodes begin with text attributes, while agent nodes begin with empty hidden states.
  • Recap: message passing in standard GNN: The framework generalizes GNN message passing through hidden states, message generation, and aggregation over local neighborhoods.Standard GNNs initialize node states from input features and iteratively aggregate neighborhood information.
  • Message passing in TextGNN: TextGNN handles agent-agent, agent-data, and data-data interactions within a unified text-based message-passing process.Different interaction types are incorporated into the general aggregation mechanism.
  • Message passing in TextGNN: For agent nodes, neighboring agent functions transform text states into messages, while data-node states are updated by aggregating messages from neighboring agents.The corresponding formulations use concatenated text inputs and task-specific agent functions.

5. RESEARCHTOWN: Applying TextGNN to Community Graph

RESEARCHTOWN applies TextGNN to a community graph in which researchers and papers are connected by authorship, expertise, and citation relations. It simulates reading, writing, and reviewing as sequential graph-inference stages with standardized outputs.

  • Inputs and outputs of RESEARCHTOWN: Task-specific inputs and standardized outputs support evaluation: abstracts are used for reading and writing, full papers for reviewing, and outputs are formatted as bullet-point summaries or critiques with scores.The hidden states condense research materials into standardized bullet points for downstream evaluation.
  • Agent-data graph for research community modeling - community graph: The community graph represents researchers as agent nodes and papers as data nodes, with citation, authorship, and reviewing-expertise edges.Researcher collaboration is inferred through two-hop paths because direct agent-agent edges are omitted.
  • Stage 1: Paper reading: Paper reading creates a new researcher profile by aggregating information from papers connected through authorship relations.The new agent starts without a profile, so this stage creates the agent representation from its paper neighborhood.
  • Stage 2: Paper writing: Paper writing creates a new paper data node from author information and citation relations after the reading stage.The paper node has no contents before writing and is generated through a TextGNN layer.
  • Stage 3: Review writing: Review writing evaluates an existing paper using reviewers rather than authors and produces the final reflection stage of the simulation.Unlike paper writing, review writing operates on a paper whose contents are already present.
  • Summary: RESEARCHTOWN simulation algorithm: RESEARCHTOWN is summarized as a two-layer GNN: paper reading performs the first aggregation layer, while paper and review writing generate the final outputs in the second.The algorithm returns paper content and review content for the target paper.

6. Evaluating RESEARCHTOWN via Masked Node Prediction Task

RESEARCHTOWN evaluates research simulation by masking nodes in a community graph and predicting their text-form outputs from neighborhood information. Real-world papers and reviews serve as ground truth for similarity-based assessment of paper writing and review writing.

  • Masked node prediction evaluates whether neighborhood information can reconstruct a hidden paper or review output.The task sets a node's hidden state to empty, predicts it from its neighborhood, and compares the prediction with the original real-world result.
  • Paper-writing evaluation treats the predicted hidden state h_v as the output for a masked ground-truth paper node.
  • Review-writing evaluation treats the predicted review r_v as the output for a masked review, whose original text is represented as r*_v.
  • Real-world papers and reviews are used as ground truth because the objective is to simulate the research community rather than optimize papers or reviews.
  • The framework updates review-generation states during evaluation to avoid accumulated errors in review-writing simulation.

7. Experimental Settings

Experiments implement RESEARCHTOWN with a deterministic GPT-4o-mini backbone and compare neighborhood aggregation strategies. RESEARCHBENCH contains paper- and review-writing tasks from recent machine-learning conferences, evaluated with embedding-based similarity metrics across paper-writing difficulty subsets.

  • RESEARCHTOWN uses GPT-4o-mini with decoding temperature 0, while AGG-global combines the target with all neighboring agent and data nodes.AGG-self, AGG-agent, and AGG-data serve as comparison baselines for different neighborhood-information choices.
  • The experimental design compares how target, agent-neighbor, data-neighbor, and global-neighborhood aggregation affect simulation outputs.
  • Paper-writing results are reported across hard, medium, easy, and overall subsets using text-embedding-large-3 similarity metrics.
  • Review-writing results report embedding-based similarity for strengths and weaknesses, with five reviewers used for AGG-agent and AGG-global.The review table also reports ΔS as the average difference of review scores.
  • RESEARCHBENCH contains 1,000 paper-writing tasks and 200 review-writing tasks sourced from recent top-tier machine-learning conferences.Paper-writing tasks are divided into hard, medium, and easy subsets with 333, 334, and 333 tasks, respectively.

8. Core Results: In-distribution Evaluation

On RESEARCHBENCH’s in-distribution cases, RESEARCHTOWN produces realistic paper-writing and review-writing outputs, with quality affected by aggregation inputs, agent count, model choice, and reference selection.

  • 67.51 overall similarity across 1,000 paper-writing tasks demonstrates realistic outputs, while review-writing scores are lower, averaging around 51 for strengths and 47 for weaknesses.Paper writing reaches 73.79 on an easy subset; weakness identification is especially challenging, possibly because real-world reviews are noisier and more diverse.
  • Combining cited-paper data with researcher agents surpasses data-only aggregation by 2.21 points, while researchers improve difficult-case scores from 56.02 to 60.89.The result suggests researcher agents contribute multi-hop paper information, whereas cited papers are generally more effective than authors in paper-writing simulation.
  • Additional reviewers consistently improve review quality, with agent aggregation best for strengths and scores and data aggregation best for weaknesses.The pattern is attributed to related-work comparisons helping identify weaknesses and multiple reviewers providing more balanced strength assessments.
  • Selecting related-work papers raises paper-writing similarity from 66.4 to 66.7, whereas introduction-only and other-section references reduce it to 65.2 and 58.4.These results show that informative reference selection matters more than simply including all available papers.
  • Increasing agents improves both tasks, with paper-writing similarity rising from 49.0 to 52.7 when agents increase from 1 to 2 and review strength from 50.8 to 51.5 when reviewers increase from 1 to 5.The gains support robustness to agent number, although the review-writing improvement is smaller.
  • RESEARCHTOWN remains robust across evaluation settings: global aggregation is generally strongest across tested generation models, and voyage-3 reproduces the main ablation trends.Deepseek-v3 outperforms GPT-4o-mini, which outperforms Qwen-2.5-7B-Instruct overall, with GPT-4o-mini showing an exception for global review difference.

10. Discussion: RESEARCHTOWN is Effective

RESEARCHTOWN is evaluated as a research-community simulator using similarity, fine-grained quality, and human-alignment assessments, while also exposing interdisciplinary promise and practical limitations.

  • 10. Discussion: RESEARCHTOWN is Effective: Fine-grained evaluation measures topic, method, factual, claim, and application-context consistency because a single cosine similarity score can mask important problems.GPT-4o assigns 0–10 scores across the five dimensions.
  • 10. Discussion: RESEARCHTOWN is Effective: RESEARCHTOWN’s simulated papers remain below real-world articles in novelty and feasibility but are close to them, indicating a need for more coordinated agentic workflows.Table 4 evaluates 1,000 simulated papers with LLM-based scores and 40 with human evaluations.
  • 10. Discussion: RESEARCHTOWN is Effective: Human judgments correlate reasonably with LLM similarity scores at Pearson correlation 0.61, but correlate poorly for intrinsic-quality evaluations because those judgments are more ambiguous.The human evaluation validates LLM-based similarity assessment more strongly than novelty and feasibility assessment.
  • 11. Case Study: Out-of-distribution Use: RESEARCHTOWN generates interdisciplinary directions by combining expertise across fields, including links between NLP and astronomy or NLP and criminology.The examples include kinematic models for language evolution and LLM support for communities affected by mass incarceration.
  • 11. Case Study: Out-of-distribution Use: Its out-of-distribution use can produce incoherent or superficial outputs when too many disparate domains are combined, limiting real-world usefulness.The paper gives a four-domain example that strings together terminology without a clear research direction.
  • 12. Conclusion: RESEARCHTOWN models research communities as heterogeneous graphs and unifies paper reading, writing, and reviewing through TextGNN-driven inference.The framework is presented as a platform for studying research dynamics and supporting automated scientific discovery.

A. Individual Contribution

The paper assigns distinct implementation and evaluation responsibilities across contributors while emphasizing ethical safeguards, human oversight, and the limited status of generated research content.

  • A. Individual Contribution: Contributors divide leadership, system design, coding, benchmark collection, experiments, paper writing, case studies, and advising across the project.The listed roles include overall project leadership, co-lead responsibilities, participation, and project advising.
  • B. Ethical Concern: The framework is designed as an assistive tool that provides inspiration instead of complete ready-to-use content, reducing risks associated with research plagiarism.Its outputs are starting points for further intellectual effort rather than replacements for researchers.
  • A. Individual Contribution: Research agents use publicly available, properly cited academic papers and are designed as domain experts rather than simulations of human dialogue or individual research styles.The stated objective is to simulate research activities using existing academic literature within limited domains.
  • B. Ethical Concern: RESEARCHTOWN outputs are intended as internal, preliminary assistance rather than direct publication, with human researchers responsible for refinement, validation, and final submission.Generated papers omit critical sections, and generated reviews remain supplementary rather than definitive.
  • A. Individual Contribution: The project documents model, data, and licensing choices, including public RESEARCHBENCH papers under CC-BY 4.0 and differing licenses for the foundation models.Inputs and outputs for RESEARCHBENCH are logged and made accessible.

D.1. RESEARCHTOWN aggregation setting implementation

The aggregation implementation varies which agent and data neighbors contribute to message passing, supports single- and multi-agent writing or reviewing, and reduces computational cost through parallelization.

  • D.1. RESEARCHTOWN aggregation setting implementation: AGG-self excludes neighborhood information, so paper writing and review writing rely only on the target content or paper.The setting uses a single LLM agent without external data or other agents.
  • D.1. RESEARCHTOWN aggregation setting implementation: AGG-agent uses only agent nodes, enabling multiple agents to share content, insights, and critiques while producing the final paper or review.Data nodes are excluded from this setting.
  • D.1. RESEARCHTOWN aggregation setting implementation: AGG-data uses only data nodes, whereas AGG-global includes both agent and data neighbors during aggregation.These settings differ according to the neighborhood nodes participating in message passing.
  • D.1. RESEARCHTOWN aggregation setting implementation: In multi-agent writing and reviewing, parallel outputs are aggregated into a final paper or review after agents reference related data sources and, where applicable, one another.Paper writing integrates agent and data insights; review writing combines perspectives on the paper and related works.
  • D.2. RESEARCHTOWN simulation algorithm implementation: The implementation reduces f_a(·) calls from N × M to M by parallelizing over data nodes, improving scalability while preserving the aggregation definition.M denotes agent nodes in neighborhoods and N denotes data nodes.

D.3. RESEARCHTOWN agent function implementation

RESEARCHBENCH constructs paper- and review-writing tasks from publicly available papers, authors, citations, and reviews, then evaluates simulation under standard and extreme-impact settings.

  • RESEARCHBENCH contains 1,000 paper-writing tasks, 100 high-impact-paper tasks, and 200 review-writing tasks.PAPERBENCH and HIGHIMPACTPAPERBENCH evaluate paper writing, while REVIEWBENCH evaluates review writing.
  • The benchmark filters for complete paper, author, citation, and review information before sampling evaluation tasks.Paper data comes from arXiv and author records are matched through Semantic Scholar; public reviews are collected primarily from ICLR 2024 via OpenReview.
  • Paper-writing tasks are divided into hard, medium, and easy subsets using simulation similarity scores, with harder papers tending to be more theoretical and mathematical.The three subsets contain 333, 334, and 333 tasks, respectively.
  • HIGHIMPACTPAPERBENCH tests simulation on the 20 most-cited papers from each of 10 leading AI conferences.It is intended as an extreme benchmark for simulating impactful research ideas.

F. Embedding-based Evaluation Details

The embedding-based evaluation decomposes papers and reviews into aligned semantic units, enabling scalable similarity measurement despite varied wording and implementation.

  • Point-wise decomposition addresses the failure of direct cosine similarity to capture conceptual equivalence across different descriptions or implementations.The same prompting structure aligns semantically related papers and reviews into comparable key points.
  • The framework replaces costly expert evaluation with semantic similarity over decomposed representations and can select the best output among multiple samples.Its format can also be extended with domain-specific dimensions such as algorithmic structure or theoretical results.
  • A VoyageAI similarity of 0.8244 between two independently written papers with nearly identical ideas provides a reference point for strong idea overlap.The authors suggest that scores above 0.82 can potentially indicate strong overlap.
  • Paper similarity is evaluated by converting generated and reference papers into aligned answers to five research questions before embedding comparison.The questions cover the problem, importance, difficulty, prior gap, and approach or results.
  • Review similarity uses bullet-point representations of strengths and weaknesses, with separate scores for each category and a score-difference measure.The metric emphasizes whether generated reviews recall points present in real-world reviews.

G. LLM-based Evaluation Details

The LLM-based evaluation adds fine-grained judgments of similarity, novelty, and feasibility, while additional experiments examine robustness, context, and high-impact-paper difficulty.

  • LLM prompting evaluates six complementary alignment dimensions, including topic, method, factual, claim, application-context, and overall semantic consistency.These dimensions capture distinct aspects of similarity between generated and reference proposals.
  • Novelty and feasibility assessments complement similarity by evaluating originality and practicality of generated research proposals.Similarity alone does not fully represent these intrinsic qualities.
  • Ablations report consistently high-quality simulations across model and hyperparameter settings, although the paper-writing ablation uses a subset rather than all 1,000 tasks.The researcher-number ablation uses 172 hard paper-writing tasks, while the review-writing ablation uses all 200 review tasks.
  • PAPERBENCH results show consistently better performance with richer AGG-data and AGG-global contexts than with AGG-self and AGG-agent contexts across three models.The comparison highlights the importance of contextual information for similarity evaluation.
  • On REVIEWBENCH, voyage-3 embeddings produce higher strength scores and larger ∆S values than text-embedding-3, while Qwen-2.5-7B-Instruct shows larger deviations from human scores.The deviations suggest potential scoring bias or overconfidence in that model’s outputs.
  • Under extreme high-impact-paper conditions, similarity is lower for groundbreaking methods and novel topics but exceeds 0.8 for some analysis and toolkit papers.Examples include Layer Normalization and Energy and Policy Considerations for Deep Learning in NLP versus Is BERT Really Robust? and Stanza.

K. Additional Case Study

Additional case studies document evaluation prompts, in-distribution outputs, and interdisciplinary generations spanning combinations of language modeling, astronomy, biology, systems, mathematics, and criminology.

  • Case-study tables present paper-writing examples across easy, medium, hard, and high-impact settings, alongside review-writing examples.These examples illustrate the evaluation process and generated outputs for PAPERBENCH, HIGHIMPACTPAPERBENCH, and REVIEWBENCH.
  • The appendix provides prompt templates for paper reading, paper writing, review writing, and LLM-based similarity and quality assessment.Separate templates cover review strengths, weaknesses, scores, and aggregation operations.
  • Interdisciplinary examples combine LLM research with astronomy, criminology, biology, systems, and mathematics in generated research proposals.Examples include astrophysical models for language evolution and tools supporting communities affected by mass incarceration.
  • Broader combinations can produce terminology-heavy or incoherent ideas when the participating domains are conceptually distant or too numerous.The reported cases include weak criminology integration, questioned practicability, and unclear practical direction.
Loading 2412.17767v2…