PaperlayerView exact revision on arXiv

Source-linked AI summary

Science Utopia? Closed-Loop LLM Simulation of Academic Research Ecosystems

Yiqiao Jin, Yiyang Wang, Lucheng Fu, Bing He, Siheng Xiong, Yijia Xiao, B. Aditya Prakash, Josiah Hester, Srijan Kumar, James Evans, Jindong Wang

arXiv:2610.01257v1cs.CLcs.AIcs.CY

TL;DR

Because research activities and institutions co-evolve, AI’s effects on individual tasks may not predict ecosystem-level outcomes. SciUtopia simulates these processes in a persistent, closed loop, finding that resubmission amplifies reviewer burden, cautious exploration balances impact and career outcomes, and resources can concentrate without a significant career advantage from narrow early funding wins.

  • Problem

    Research activities, networks, resources, and the scientific record co-evolve, making it difficult to trace ecosystem effects or observe counterfactual policy outcomes in real academia.

    Source: §1 · pp. 1–2

  • Method

    SUTO is a persistent, closed-loop LLM simulation integrating research activities and supporting matched counterfactual experiments under shared simulation rules.

    Source: §1 · p. 2

  • Results

    Resubmission amplifies reviewer burden beyond population growth, cautious exploration balances citation impact with career outcomes and topic diversity, and resources concentrate despite no significant career advantage from narrow early funding wins.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    SUTO provides a testbed for understanding how local decisions and academic rules accumulate into long-term scientific dynamics.

    Source: §5 · p. 10

    SUTO does not simulate proposal funding, assuming publications are positively associated with proposal-review evaluations and funding outcomes.

    Source: §3.1 · p. 5

Abstract

from arXiv · show

Scientific progress emerges from a longitudinal ecosystem in which researchers, institutions, funding agencies, collaboration networks, and the scientific literature co-evolve. As AI becomes increasingly involved throughout the scientific research cycle, understanding these interconnected and evolving processes becomes increasingly important. We introduce SciUtopia, a persistent, closed-loop LLM-agent simulation framework for studying academic research ecosystems. SciUtopia models interconnected scientific processes such as research-direction choice, collaboration, submission, peer review, resubmission, citation, funding, and researcher attrition, while maintaining evolving states across simulated years. Its configurable institutional mechanisms and information channels provide a controlled testbed for matched counterfactual experiments and targeted interventions. Across 61 simulation worlds, SciUtopia simulates over 40,000 researchers from 8,000 institutions, producing around 400,000 publication decisions and 1.2 million LLM-generated peer reviews. Using these longitudinal simulations, we find that rejection-driven resubmission substantially amplifies reviewer burden beyond population growth alone, cautious exploration balances citation impact with career success and long-term topic diversity, and resource inequality can emerge even without detectable cumulative advantage from narrowly winning early funding. Code is available at https://github.com/Ahren09/ScienceUtopia.

1 Introduction

AI-mediated research decisions feed back through an evolving ecosystem, so isolated improvements need not produce a more productive, diverse, or sustainable system. SUTO is introduced as a controlled, longitudinal simulation for studying these dynamics and their consequences.

This Work

  • SUTO integrates interconnected research processes into a persistent simulation supporting matched counterfactual experiments across 61 worlds, 40,000-plus researchers, and 8,000 institutions.The framework follows publication decisions and peer reviews as well as researcher states and the shared scientific record.Source: §1 · p. 2

Findings

  • Resubmission alone raises review demand to 6.64 reviews per active reviewer, versus 2.07 vs. 2.25 under population growth alone; stricter acceptance criteria add 14% workload.Resubmissions sustain manuscript backlogs and shift review work into future rounds.Source: §1 · pp. 2–3
  • Doubling the submission limit raises accepted output 63–69% in university-only worlds, yet under fixed grant capacity year-ten participation falls from 39% to 24%.Long-term participation also declines when funding capacity scales with demand.Source: §1 · p. 3
  • Cautious explorers gain greater citation impact than exploiters and explorers, with only modest declines in acceptance and funding; widespread cautious exploration keeps topic activity broadly distributed.The strategy explores new topics close to researchers’ expertise.Source: §1 · p. 3
  • Narrow early funding wins show no significant career advantage over the next three years, while resources become concentrated and nearly half of researchers exhaust them.Source: §1 · p. 3
  • Revealing the first author’s pseudonymous name and institution raises review scores by 0.055–0.077 points on a 1–5 scale; further network disclosure adds little.The additional disclosure has little effect once author information is visible.Source: §1 · p. 3

2 The SUTO Framework

SUTO models research ecosystems as a persistent, closed-loop process in which research, review, publication, and funding outcomes shape later decisions. Its researcher states, institutional settings, and yearly pipeline represent these interactions over time.

2 The SUTO Framework

  • SUTO simulates research, peer review, publication, and funding as a closed loop in which outcomes shape decisions across simulated years.The framework also includes direction choice, collaboration, citation, resubmission, and researcher attrition.Source: §2 · p. 3

2.1 Researchers, Institutions, and Venues

  • Researchers carry expertise, reputation, budget, conflicts of interest, and memory that links past reviews and publication outcomes to later decisions.Exhausting the budget makes a researcher inactive; budgets reflect funding received and action costs.Source: §2.1 · p. 3
  • The framework represents explorers, cautious explorers, and exploiters alongside distinct funding programs and venues grounded in a temporally restricted literature corpus.Researchers select compatible venues and retrieved papers for submission, and accepted papers can be cited in later years.Source: §2.1 · pp. 3–4

2.2 Simulation Pipeline

  • Each simulated year runs research direction and collaboration, submission and citation, peer review, decisions and memory updates, funding and attrition, then resubmission.Resubmission begins in year two; rejected papers may be abandoned, revised, or resubmitted based on reviews, submission history, resources, and cost.Source: §2 · p. 3, §2.2 · pp. 4–5

3 Experiments and Findings

The experiments examine how research strategies, publication and funding mechanisms, and ecosystem dynamics shape careers and scientific outcomes. They find that resubmission raises reviewer workload, cautious exploration supports career outcomes and topic diversity, and resource inequality can coexist with broad research diversity.

3.1 Human Factors: Exploration Strategies and Careers

  • Cautious explorers trade modestly lower acceptance and funding than exploiters for stronger citation impact, while outperforming distant explorers on several career outcomes.Explorers had the lowest acceptance rate (29.8%), versus 37.2% for cautious explorers and 39.2% for exploiters; by year ten, 41.1% of explorers remained active, compared with about 53% of cautious explorers and exploiters.Source: §3.1 · p. 6
  • Cautious-heavy worlds maintain higher topic entropy and more active research directions than exploiter-heavy worlds, and exceed explorer-heavy worlds in ten-year entropy AUC.Across matched counterfactual sets, cautious-heavy populations maintain 0.11 higher normalized topic entropy than exploiter-heavy populations and end with 8.5 more active research directions; their ten-year entropy AUC exceeds explorer-heavy worlds by 0.156 on average.Source: §3.1 · p. 6, Figure 3 · p. 6

3.2 Research Outcomes: Impact, Resubmission, and Peer Review

  • Across ten simulated years, population growth alone leaves per-reviewer workload close to baseline, whereas enabling resubmission produces substantially higher workload; together, population growth and resubmission compound submission volume.Source: Figure 5 · p. 8
  • Greater topical departure is associated with lower acceptance but higher citations among accepted papers; it does not generally correspond to more disruptive papers.For cautious explorers, the share of disruptive papers rises from 47.6% to 57.7% between intermediate and high departure, while it falls from 63.6% to 41.2% for exploiters.Source: §3.2 · p. 7, Figure 4 · p. 7
  • Author-information disclosure raises LLM review scores slightly in the reported comparisons.Disclosure raises scores by 0.055 points on the 1–5 scale; a prespecified 120-paper follow-up finds a 0.077-point increase (95% CI [0.026, 0.128]).Source: §3.2 · p. 8

3.3 Institutional Mechanisms: Funding Capacity and Allocation

  • Doubling submission intensity increases accepted papers but reduces long-run participation; fixed grant capacity also leaves more researchers without awards, with only modest changes in recipient inequality.In university-only worlds, year-ten active shares fall from 49% to 29% under scaling capacity and from 39% to 24% under fixed capacity; under fixed capacity, the never-funded share rises from 22% to 37% with one submission per project and from 24% to 46% with two.Source: Figure 6 · p. 8, §3.3 · p. 9

3.4 Emergent Ecosystem-Level Dynamics

  • Across three decade-long confirmatory worlds, funding inequality rises while topic entropy stays high and all 53 research directions attract active researchers, alongside substantial attrition.The funding Gini rises from about 0.04 to about 0.36; the active population falls from 5,000 to approximately 2,530 by year ten.Source: §3.4 · p. 9

4 Related Work

Prior work spans LLM systems for individual research activities and agent-based models of science. SUTO couples these processes in a persistent, literature-grounded ecosystem to study how local mechanisms propagate into longer-run outcomes.

Large language model agents and agent-based models of science

  • LLM systems have been applied to idea generation, experimentation, manuscript preparation, and peer review.The related work contrasts these task-focused systems with SUTO's coupled ecosystem simulation.Source: §4 · p. 10
  • Agent-based models of science have studied incentives and institutional rules across publication, reviewing, funding, cumulative advantage, and adversarial participation.SUTO combines research choice, collaboration, publication, peer review, resubmission, citation, funding, resources, and career survival in a persistent ecosystem.Source: §4 · p. 10

Large language model-based social simulation

  • LLM-based social simulations offer controlled tests of complex systems, while newer research-community simulations model repeated agent interactions or researcher-paper graphs.The passage identifies AgentRXiv and ResearchTown as examples of these community-level approaches.Source: §4 · p. 10

5 Conclusion

The simulations show that repeated resubmission raises reviewer demand much more than population growth alone, while the conclusion presents SUTO as a testbed for studying how academic rules shape evolving research ecosystems. The supplied passages also report strategy-manipulation checks, define long-run diversity measures, and qualify disruption results by citation uptake.

5 Conclusion

  • SUTO integrates research, publication, funding, memory, and attrition in a persistent loop to study how local academic rules shape long-term ecosystem dynamics.The framework supports matched counterfactual experiments that vary an institutional mechanism under the same simulation rules.Source: §5 · p. 10

A.1 Cautious exploration balances novelty with career reward

  • Confirmatory worlds preserve the strategy ordering in exploration distance: explorers averaged 0.83, cautious explorers 0.69, and exploiters 0.49.These worlds used 1,000 institutions over ten years, with five researchers per institution and initial strategy composition of one explorer, three exploiters, and one cautious explorer.Source: §A.1 · p. 13

A.2 Cautious exploration sustains long-run scientific diversity

  • The strategy-proportion study compares ten-year normalized topic-entropy area, occupied research directions, and total submissions across matched strategy compositions.It uses ten matched triples of 100-institution worlds with explorer-heavy, exploiter-heavy, or cautious-heavy compositions.Source: §A.2 · p. 13

B.1 Disruption Depends on Uptake

  • At age three, 98.5% of papers with CD = 0 were uncited, and cited papers from all three strategies were similarly disruptive.Consolidating papers never exceeded 2.0% of a departure bin; CD was reported only for papers with references and at least three papers in its denominator.Source: Figure 7 · p. 13, §B.1 · p. 13

B.2 Submission growth and reviewer workload under population change and resubmission

  • 2.07 vs. 2.25 reviews per active reviewer under population growth alone contrasts with 6.64 under resubmission alone, while resubmission also raises year-ten activity attrition.With both mechanisms enabled, workload is 5.48 reviews per active reviewer; by year ten, 221 of 302 researchers remain active under resubmission versus 257 without it.Source: §B.2 · p. 14

B.3 Additional Resubmission-Cascade Results

Figure 8 tracks how resubmission multiplies submissions and how acceptance conditions relate to resubmission cascades.

B.3 Additional Resubmission-Cascade Results

  • 2.78 without population growth and 2.59 with growth are the year-ten submission multipliers, where the multiplier is total submission events divided by first submissions.The dotted reference line marks a multiplier of 1 without resubmission.Source: Figure 8 · p. 15
  • Under fixed 30% versus declining ACL-calibrated acceptance, panel (b) compares mean venue attempts for papers rejected at least once and the fraction resubmitted.The bars show paper-level bootstrap 95% confidence intervals.Source: Figure 8 · p. 15
  • Panel (c) compares mean prior rejections among papers accepted after resubmission in each year under fixed and declining acceptance rates.The shading shows the mean plus or minus 1.96 standard errors; each condition is one world, so intervals do not estimate across-world variability.Source: Figure 8 · p. 15

B.4 No-influx resubmission comparison

In the no-influx comparison, the resubmission condition has substantially more reviews and higher normalized review demand than the condition without resubmission. The comparison is descriptive and uses one ten-year world per condition.

B.4 No-influx resubmission comparison

  • The matched conditions use the same 1,202 founders and differ in whether resubmission is enabled; neither admits new researchers.Review demand is measured as annual reviews divided by the year-end active pool and also normalized by the initial population; the annual multiplier is total submissions divided by first submissions.Source: §B.4 · p. 15, Figure 9 · p. 16
  • 3,660 versus 1,608 individual reviews in year ten distinguish C, with resubmission, from A, without it; active-pool ratios are 5.666 versus 1.926 reviews.On the full 1,202-person pool, the ratios are 3.045 versus 1.338; C has fewer first submissions (432 versus 536) plus 788 repeats.Source: §B.4 · p. 15, Figure 9 · p. 16
  • Across ten years, C records 13,952 submission events and 41,856 reviews, versus 6,566 events and 19,698 reviews in A.Each submission event receives three reviews, so review totals are mechanically linked to submission totals rather than independent evidence of research production or quality.Source: §B.4 · p. 15
  • One world per condition provides no across-world uncertainty estimate, and the comparison does not establish that reviewing caused inactivity.The authors characterize it as a descriptive review-demand contrast, not an influx interaction or an isolated funding-mechanism test.Source: §B.4 · p. 15

B.5 Details of the Author-Information Experiment

The author-information experiment compares blinded review with disclosure of author identity and network relationships while controlling the paper and reviewer conditions. A separate output-scaling study defines its worlds, interventions, and descriptive outcome measures.

B.5 Details of the Author-Information Experiment

  • Each reviewer evaluates the same paper under blind review, author-information disclosure, and author information plus a neutral description of the network relationship.The paper, topic labels, reviewer expertise, scoring instructions, and 1–5 scale are held fixed; the matched reviewers differ in their relationship to the first author.Source: §B.5 · pp. 16–17
  • 0.055 points (95% CI [0.014, 0.097]) was the mean score increase from revealing author information, while adding relationship disclosure after that changed scores by 0.000 points (95% CI [−0.038, 0.037]).Reviewers sharing a collaborator with the author did not show a detectably larger response to relationship disclosure than stranger reviewers.Source: §B.5 · p. 17

C.1 Higher research output expands publication volume faster than ecosystem capacity

  • The output-scaling study uses six ten-year worlds to vary submission limits and funding capacity, while measuring participation, grant coverage, citations, inequality, and topic entropy.Citation measures are descriptive because citation opportunities also depend on submission volume and accepted-paper-pool size.Source: §C.1 · p. 17

C.2 Narrow funding wins show no detectable increase in later funding

Near-cutoff funding wins show no detectable career advantage over three years, while funding resources become more concentrated as topic diversity holds across the confirmatory worlds.

C.2 Narrow funding wins show no detectable increase in later funding

  • No detectable near-cutoff differences appear in later funding, publications, citations, future award probability, or exploration over three years.The estimated funding effect is −1.50 resource units (95% CI [−14.5, +11.5]).Source: Figure 10 · p. 18, §C.2 · p. 18
  • Applicants just above versus below panel funding cutoffs provide the local comparison; funding estimates remain indistinguishable from zero across bandwidth checks.The study follows 444 applications from 265 applicants in 183 competitions; estimates across h = 2, 3, 4, 6, 8 range from −1.50 to +0.47, with intervals spanning zero.Source: §C.2 · p. 18

D.1 Resource Inequality and Researcher Attrition

  • Across three confirmatory worlds, funding-resource inequality rises while normalized topic entropy holds; active researchers decline over time.Mean review scores remain between 2.29 and 2.40 on the 1–5 scale, with a median of 2.0 each year.Source: Figure 11 · p. 19

D.2 Self-citation does not explain citation inequality

Self-citations are too rare to explain citation inequality in the simulated ecosystem, despite differing in frequency across research strategies.

D.2 Self-citation does not explain citation inequality

  • Exploiters self-cite in 0.57% of outgoing edges, versus 0.13% for cautious explorers and 0 of 10,861 edges for explorers.The ordering holds in all three worlds and is consistent with topical continuity, but self-citation is too rare to explain citation concentration.Source: §D.2 · p. 19
  • Removing self-citation edges changes citation Gini by at most 0.0005 in any world; self-citations account for 308 of 75,661 edges (0.41%).Across the three confirmatory worlds, exploiters self-cite more frequently than peers.Source: Figure 12 · p. 19, §D.2 · p. 19

E Notation

F.1 Data

The simulations ground their scientific literature in SciEvo and retrieve temporally eligible papers to support submissions and citations.

F.1 Data

  • SciEvo contains 2.1 million arXiv papers from 1991–2025 across eight subject groups and 156 categories, with metadata, keywords, citation relations, authors, and venues.The experiments select computer science papers from the 2016–2025 simulation window and organize them by publication year.Source: §F.1 · p. 21

F.2 Model and literature grounding

  • Submissions retrieve contemporaneous arXiv papers from SciEvo by topical compatibility, while citations retrieve previously accepted papers in the same world.The simulations use Qwen3-32B in thinking mode; unless otherwise specified, conferences use a fixed 30% acceptance rate.Source: §F.2 · p. 21
  • All model families use the same set of research directions, D.Source: §F.2 · p. 21

F.3 Implementation settings

The implementation carries researchers and scientific records forward through annual feedback loops, enabling institutional interventions to alter later ecosystem states. The reported simulations also provide a large corpus and illustrate system-level outcomes and research-strategy patterns.

F.3 Implementation settings

  • Resource settings, capacity treatments, population compositions, and world counts are specified for the reported studies.Table 2 records resource and funding settings; budgets and costs use abstract resource units.Source: §F.3 · p. 21, Table 2 · p. 21

F.4 Motivation for closed-loop simulation

  • Updated researcher states and scientific records feed later decisions, while institutional rules govern quotas, resource accounting, and attrition.Rejected manuscripts remain available for resubmission; accepted papers enter the citable literature and authors’ funding-evaluation records.Source: §F.4 · p. 22, Figure 13 · p. 22

F.5 Simulation corpus and released dataset

  • Across 61 worlds, SUTO simulates over 40,000 researchers at 8,000 institutions over ∼390,000 researcher-years, producing ∼400,000 publication decisions and ∼1.2M reviews.The runs also record ∼177,000 submitted manuscripts and ∼595,000 researcher memory entries.Source: §F.5 · p. 22

G Additional Figures

  • The annual simulation cycle has six phases, with each phase updating persistent researcher states for subsequent decisions.Source: G Additional Figures · p. 23
  • For cautious explorers, citation gains rise most consistently with topical departure; the relationship is weak for explorers and exploiters.Figure 4 plots accepted papers by topical departure from recent work and citations within three years.Source: G Additional Figures · p. 23

H Broader Impact

  • The framework supports counterfactual experiments on institutional mechanisms whose long-term effects are difficult to isolate in real scientific systems.The paper presents ecosystem simulation as a complementary tool for studying how institutional mechanisms shape science’s long-term trajectory.Source: H Broader Impact · p. 23
Loading 2610.01257v1…