PaperlayerView exact revision on arXiv

Source-linked AI summary

IdeaTaste: Do LLMs Share the Research Taste of Human Researchers?

Shanshan Zhong, Chen Xu, Jingjie Ning, Zhengzhong Liu, Aviral Kumar, Chenyan Xiong

arXiv:2611.03424v1cs.CL

TL;DR

Research taste—what to read, which problems to pursue, and which methods to try—remains less understood than AI scientists’ ability to execute research. IdeaTaste compares proposals based on literature frozen through 2025 with papers those communities publish in 2026. Across more than 90,000 proposals, agents fall well short of human communities, especially in matching ideas.

  • Problem

    Research taste is increasingly consequential as AI scientists take on more research work, but remains less understood than their ability to execute it.

    Source: §1 · p. 1

  • Method

    IdeaTaste benchmarks proposals generated from literature frozen through 2025 against 2026 papers from the same communities, measuring matching and diversity in citations and ideas.

    Source: §1 · p. 2

  • Results

    Across more than 90,000 proposals, agents reach about 26% of the human level on citation matching and only 18% on idea matching.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Agents track community directions more readily than they produce its concrete approaches, making how to pursue a direction the central challenge identified for future AI scientists.

    Source: §6 · p. 10

Abstract

from arXiv · show

Research taste, the judgment of what to read, which problems to pursue, and which methods to try, increasingly shapes the work of AI scientists, yet it remains far less understood than their ability to execute research. We introduce IdeaTaste, a benchmark that scores an LLM by how closely the research proposals it writes from a literature snapshot frozen at 2025 match the work the same communities published in 2026. Our experiments span six contemporary language models and four research areas from established to emerging fields, covering more than 90{,}000 proposals in total. LLM agents fall well short of human communities in both the literature they cite and the ideas they state, reaching about 26\% of the human level on citation matching and only 18\% on idea matching. Their populations can cite as diversely as a community and take up its tasks and methods, yet rarely arrive at the concrete approaches and research plans it pursues. Neither richer field context nor larger populations substantially close this gap. In blinded expert review, AI proposals trail the 2026 papers least in presentation and most in expected success, echoing their gap in concrete approaches, and those whose ideas no human paper shares fare worst, suggesting weaker directions rather than overlooked opportunities. These results indicate that the main bottleneck in AI research taste lies less in finding the directions a community works on than in turning them into the concrete approaches it goes on to pursue. We release IdeaTaste as an open, continuously updated leaderboard (https://cxcscmu.github.io/IdeaTaste) that benchmarks contemporary AI proposals against future publication cycles.

1 INTRODUCTION

IdeaTaste compares proposals written from a literature snapshot frozen through 2025 with what the same human research communities publish in 2026. It tests whether LLMs share human research taste, a choice-making capability increasingly relevant as AI scientists take on more research work.

1 INTRODUCTION

  • Existing evaluations assess ideas individually or against fixed targets, while population studies measure repetition rather than resemblance to human research choices.The introduction identifies direct comparison with the research choices of the same human community as the missing evaluation.Source: §1 · p. 1
  • IdeaTaste uses 2026 papers from the same communities as a reference for evaluating proposals written from a literature snapshot frozen through 2025.It also uses blinded expert review to test whether proposals unmatched by 2026 papers are overlooked opportunities or weaker ideas.Source: §1 · p. 2, Figure 1 · p. 2
  • The benchmark measures agreement with human papers and proposal diversity at both the citation and idea levels across four research areas.It tests six language models under four methods that vary what agents know about their area.Source: §1 · p. 2
  • Across more than 90,000 proposals, agents reach about 26% of the human level on citation matching and only 18% on idea matching.No population reaches half the human level on either metric; the shortfall is greater in proposed ideas than in cited literature.Source: §1 · p. 2
  • Agents can take up a community’s tasks and methods, yet their proposals match human papers more often on problems and motivations than on concrete approaches and research plans.Blinded reviewers rate AI proposals least below papers in presentation and most in expected success; unmatched ideas fare worst.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work studies AI research agents, idea generation, and idea evaluation, but leaves a gap between internal measures of population similarity and comparison with human research choices. IdeaTaste uses later community publications as an external reference for both what agents cite and what they propose.

AI scientists and research agents

  • Research-agent evaluations have largely focused on single systems, while IdeaTaste compares a population of agents proposing from shared literature.The related work describes systems for executing ideas, refining hypotheses, and conducting multi-agent research across scientific domains.Source: §2 · p. 3

Automated idea generation

  • Population-scale idea-generation studies find similarity within and across models, but mostly compare agents with one another rather than with human research.IdeaTaste measures similarity in cited literature and stated ideas and pairs both with agreement against human papers.Source: §2 · p. 3

Evaluating research ideas

  • Idea evaluations use expert ratings, fixed-target benchmarks, or model judges, while published research offers a later reference for community choices.IdeaTaste extends prior uses of published work from individual papers to what a community publishes after its literature is frozen.Source: §2 · p. 3

3 IDEATASTE

IdeaTaste compares proposals generated from a frozen literature corpus with subsequent papers from the same research areas. It evaluates both citation choices and idea formulation, and varies the information available to agents.

3.1 DATA CONSTRUCTION

  • The benchmark builds a frozen literature corpus and a later human reference, then compares proposals against what the community publishes.The corpus includes papers through 2025; the 2026 reference covers papers collected through August, and its collection end date can move forward.Source: §3 · p. 3, §3.1 · p. 4, Table 1 · p. 4
  • The corpus and human reference are assembled by following citations from area-specific landmark papers, with topic search also used for the 2026 reference.The corpus adds papers cited by at least two landmark-citing papers; the human reference is screened against area definitions and excludes surveys.Source: §3.1 · p. 4

3.2 EVALUATION DESIGN

  • Four metrics compare human papers and proposals on citation matching, citation diversity, idea matching, and idea diversity against a human-paper baseline.Documents are represented by citations from the frozen corpus and by summaries extracted into five idea slots: contribution type, problem, motivation, proposed approach, and research plan.Source: §3.2 · p. 4

3.2.1 LEVEL 1: LITERATURE SELECTION

  • MATCH-CITE uses citation-set Jaccard similarity and requires both proposals to resemble human papers and human sources to appear across proposals.Citation diversity is high only when proposals cite enough and read different sources; its sufficiency term is capped at the mean human citation-set size.Source: §3.2.1 · pp. 4–5

3.2.2 LEVEL 2: IDEA FORMULATION

  • Idea metrics compare weighted matches across five slots, with rarer slots receiving greater weight.MATCH-IDEA measures the weighted fraction of human slots covered by proposals; DIVERSITY-IDEA measures one minus the weighted fraction of slots shared with proposal peers.Source: §3.2.2 · p. 5

3.3 METHODS

  • All methods share an agent implementation but vary its area knowledge, from no explicit prior to topics, a landscape of area structure, or peers’ earlier proposals.Peer records in FRONTIER are separate from literature retrieval, cannot be cited, and are marked as not scientifically validated.Source: §3.3 · p. 5

4 EXPERIMENTS

Across more than 90,000 proposals, LLM agents match human communities far less on ideas than citations. The analyses locate the gap in concrete approaches and research plans, and blinded reviewers rate proposals—especially those without human counterparts—less favorably on execution-related dimensions.

Experimental setup

  • The evaluation uses six models, four methods, and populations of 300 proposals; blinded review covers 192 documents assessed by 11 experienced reviewers.Four models were released in 2026 and may have seen the reference in training; two pre-2026 models could not have seen 2026 papers.Source: §4 · pp. 6–7

Main results

  • 26% of the human level on MATCH-CITE and 18% on MATCH-IDEA: LLM agents fall far short, with no population reaching half the human level on either metric.The gap is narrowest in automated research and widens in machine learning systems and code generation.Source: §4 · p. 7
  • Across five of six models, matching falls further behind on ideas than citations; newer training data and richer context do not close the gap.The two pre-2026 models reach about 26% of the human level on idea matching, against 14% for the four models that may have seen the 2026 papers; matching stays within a narrow band across methods.Source: §4 · p. 7
  • In blinded review, AI proposals score 20 points below 2026 papers on expected success but only 4 points on presentation.Proposals without human counterparts fall further behind on execution: their gaps widen from 8 to 15 points in feasibility and from 15 to 26 in expected success.Source: §4 · pp. 7–8, Table 3 · p. 7
  • The findings indicate that agents more readily reach community research directions than the concrete approaches they pursue, and proposals outside the human record read as weaker rather than overlooked.The authors report that this pattern persists across newer models, richer context, and larger populations.Source: §4 · p. 8

Human evaluation

  • Individual blind-review cases vary: one matched proposal scores higher than its paired paper on soundness and expected success, while unmatched documents span a range of ratings.In one unmatched case, a 2026 paper is rated below two proposals, while another proposal is mistaken for a published paper.Source: §4 · p. 8, Figure 2 · p. 8

5 ANALYSIS

The analysis finds that LLM agents often cover human research tasks and methods yet combine them differently and still miss the ideas pursued. The gap is most evident in proposed approaches and research plans, and larger populations do not remove it.

Research-direction coverage

  • On task and method axes separately, directed populations come close to human tag mass, but even FRONTIER reaches only about half the human mass on task–method pairs.FRONTIER reaches about twice the human pair mass of FREE but matches barely more human ideas.Source: §5 · p. 8, Figure 3 · p. 9

Idea-slot analysis

  • LLM agents match human problems and motivations far more often than proposed approaches and research plans, where almost no human approach or plan finds a counterpart.This ordering also appears among 2026 papers themselves; Llama 4 Scout stays at roughly a third of the human level on every slot, while DeepSeek-V4-Flash covers a small share of human problems and almost none of their plans.Source: §5 · p. 9, Table 4 · p. 9

Concrete cases

  • In the paired examples, proposals that miss the human approach keep the trained network unchanged and add machinery around it, unlike the human papers that modify the network.The examples include expert prefetching and retrieved transitions as alternatives to modifying routers or low-rank adapters.Source: §5 · p. 9, Figure 4 · p. 10

Population size

  • Matching rises only gently and levels off as populations grow from 300 proposals to 3.8–17 times that size, remaining well below the human level.TOPIC rises more steeply than FREE on ideas in most settings, but both remain far from the human level.Source: §5 · pp. 9–10, Figure 5 · p. 10
  • The analysis concludes that broader direction coverage and more proposals do not bring agents closer to the concrete approaches the community pursues.The proposed research-taste bottleneck is how a recognized problem becomes a concrete approach, rather than how widely or how much a population proposes.Source: §5 · p. 10

6 CONCLUSION

IdeaTaste benchmarks LLM research proposals against what human communities later publish. The conclusion is that agents more often share a community’s literature and directions than its concrete approaches, with proposals outside the human record tending to be weaker.

6 CONCLUSION

  • LLM agents’ research taste appears limited less by knowing what to work on than by deciding how to pursue it, a pattern the authors say holds across newer models, richer context, and larger populations.Proposals that depart from human research tend to be weaker rather than overlooked opportunities.Source: §6 · p. 10

A DATA CONSTRUCTION DETAILS

The benchmark separates a pre-2026 literature corpus used by agents from a 2026 human-paper reference used to assess proposals. Its four research areas are defined in advance, with the same area descriptions provided to agents and used in screening.

A DATA CONSTRUCTION DETAILS

  • The pre-2026 corpus supplies retrieved background, while the 2026 reference set is used to measure proposals; the two sets are built to different standards.The corpus must be wide enough to contain the background of 2026 papers and exclude 2026 work, while the reference aims for completeness within the area.Source: A DATA CONSTRUCTION DETAILS · p. 17

A.1 RESEARCH AREAS

  • The efficient ML systems area includes work on efficient training and inference, but excludes task-quality architecture work and unrelated generic high-performance computing.Its collection uses OpenAlex topics and searches for “llm serving,” “inference serving,” and “distributed training.”Source: §A.1 · p. 17, Table 5 · p. 17
  • The code-generation area covers learned production of code from specifications, including synthesis, code evaluation, repair, coding agents, and code data curation.Generic foundation-model work and papers that merely use code as an application domain are out of scope.Source: §A.1 · p. 17, Table 5 · p. 17
  • The world-model area covers learned environment models that predict future states under actions for planning, simulation, or data generation.Generic video generation, model-free reinforcement learning, and generic LLM or NLP work are excluded.Source: §A.1 · p. 17, Table 5 · p. 17
  • The automated-research area includes AI systems that automate research stages from idea generation and experimentation through writing and peer review.Generic agent frameworks without a research-automation focus and domain-science papers that merely use AI are out of scope.Source: Table 5 · p. 17, §A.1 · p. 18

A.2 DATA COLLECTION

IdeaTaste builds a pre-2026 literature corpus and a 2026 reference set for each research area, then compares collected human choices with proposals generated from the frozen corpus. The appendix details collection coverage and the evaluation procedures for citation and idea matching.

A.2 DATA COLLECTION

  • Citation expansion starts from 20–22 hand-picked landmarks: descendants citing a landmark enter the corpus, then papers cited by at least two descendants add shared background.The 2026 reference set combines citation expansion with topic search, screens papers against the area definition, and removes surveys, duplicates, and corpus overlap.Source: §A.2 · p. 18, Automated research · p. 20

Automated research

  • The reference-set snapshot covers January through August 2026, with August only partly filled.Collection combines citation expansion and topic search, screens against the area definition, and removes surveys, duplicates, and records already in the corpus.Source: Automated research · p. 20, Table 8 · p. 21

A.3 COVERAGE OF SURVEY REFERENCES

  • Automated research has the lowest survey-reference coverage, and its audit reaches furthest into 2026; a preprint posted shortly before the snapshot may miss both collection routes.The code-generation keyword-filtered corpus covers only a small fraction of the references reached by citation expansion, supporting the use of citation-based collection.Source: §A.3 · p. 21, Table 11 · p. 23

A.4 INTERPRETATION OF THE HUMAN REFERENCE

  • The 2026 papers are treated as records of human choices to pursue problems and ideas, not as proof that published work is high quality or scientifically valuable.IdeaTaste measures agreement with realized community choices; non-matching AI proposals are judged less feasible and less likely to succeed than non-matching human papers.Source: §A.4 · pp. 21–22, §A.5 · p. 22

A.6 INTERPRETING CORPUS SIZE

  • Corpus size reflects how widely an area’s work is built on elsewhere, while collection gaps include older papers without abstracts and recent or uncited preprints that may be missed.Some 2026 reference papers without parsed references remain in the reference set but are excluded from the denominator for citation resolution into the corpus.Source: §A.6 · pp. 22–23, §A.7 · pp. 23–24

B.1 CITATION SUFFICIENCY THRESHOLD

  • The per-area citation-sufficiency threshold τ is the mean number of citations in human reference papers after citations are restricted to the corpus.Because citation counts are right-skewed, fewer than half of human papers reach τ; a population that passes the threshold reads more per proposal or spreads wider than the community.Source: §B.1 · p. 24

B.2 IDEA SCHEMA AND EXTRACTION

  • A language model extracts each abstract into five idea-schema slots, records unstated content as “None,” and is instructed not to repair gaps or copy coined names or outcomes.The slots are contribution type, problem, motivation, proposed approach, and research plan.Source: §B.2 · pp. 24–25

B.3 CANDIDATE RETRIEVAL, IDEA MATCHING, AND SLOT WEIGHTS

  • Idea matching combines sparse and dense retrieval to select five candidate documents, then judges each pair by slot and whole-idea agreement; rarer human-reference slot matches receive greater weight.Slot weights are estimated once per area from the 2026 human papers and held fixed across methods.Source: §B.3 · pp. 26–27

B.4 HUMAN EVALUATION OF THE IDEA JUDGE

The study tests whether human annotators make the same slot-level and whole-idea judgments as the language-model judge. The judge’s repeated decisions are largely reproducible and align closely with human majorities on some slots.

Questionnaire

  • Annotators judge four slots independently and then make a separate whole-idea decision, using the same summaries and decision stages as the model judge.A slot left empty by either document counts as different; annotators also record confidence.Source: §B.4 · p. 27, Figure 9 · p. 28

Sampling design

  • The study samples 40 summary pairs and 160 slot pairs, stratifying across pair type and judge decisions to avoid a sample dominated by easy disagreements.Five pairs are drawn from each of eight cells; the resulting sample covers 14 of the 16 possible combinations of the four decisions annotators repeat.Source: §B.4 · p. 27

Judge consistency and agreement with annotators

  • 77–90%: judge runs agree across repetitions, with close alignment to annotator majorities on problem and proposed approach but lower alignment on motivation and research plan.On most slots, judge–human alignment matches or exceeds agreement among annotators.Source: §B.4 · pp. 27–28, Table 13 · p. 28

B.5 HUMAN EVALUATION OF IDEA QUALITY

The paper evaluates proposal quality through blinded human ratings and describes the shared loop used to generate proposals. Reviewers assess multiple dimensions, while agents plan, search, read, draft, and submit within a step budget.

B.5 HUMAN EVALUATION OF IDEA QUALITY

  • 192 documents are rated: 96 AI proposals and 96 papers from 2026, balanced across four areas, with equal numbers in four overlap categories.The sample includes 16 proposals per model, and documents are divided into six blocks of 32.Source: Table 14 · p. 29, §B.5 · p. 29
  • Reviewers score soundness, presentation, contribution, feasibility, and expected success, alongside an overall rating, confidence, and a blinding check.The dimensions use different scales, and population means are mapped linearly to [0, 100].Source: §B.5 · pp. 29–30
  • Eleven reviewers with review experience each rate assigned blocks; overlapping proposals and their paired 2026 papers are placed in different blocks.Each document receives two review records, so paired comparisons are across reviewers and the analysis includes a reviewer random effect.Source: §B.5 · pp. 29–30
  • A blinding check asks whether reviewers think each entry is a real 2026 paper, an AI proposal, or unclear; ratings and confidence use separate scales.Overall ratings range from strong reject to strong accept, while confidence ranges from unable to assess to absolutely certain.Source: §B.5 · p. 30

C.1 SHARED AGENT LOOP

  • Each agent run produces one proposal in a ReAct-style harness, where the model chooses actions and spends a budget of at most 20 calls.The shared actions are plan, search, read papers, draft, summary, and proposal.Source: §C.1 · p. 31, Figure 12 · p. 33
  • Agents work from literature ending December 31, 2025, and must ground research-gap claims in the bounded corpus rather than treat missing results as proof that work does not exist.Proposals are required to state a specific, falsifiable idea and explain how it differs from retrieved related work.Source: §C.1 · pp. 31–32

C.2 METHOD CONFIGURATIONS

The section specifies how agents receive field context, how 2025 literature is converted into a community landscape, and how models and analyses are configured. It also describes topic allocation and the prompt profiles that distinguish the agent methods.

C.2 METHOD CONFIGURATIONS

  • FREE supplies no research direction, TOPIC names an area and lets agents choose problems, and FRONTIER differs from TOPIC only through access to earlier rounds.The methods also differ in which profile appears first in the prompt and which actions are appended.Source: §C.2 · p. 33
  • TOPIC profiles allocate agents across OpenAlex topic labels, weighted by publication year with a two-year half-life, while leaving their search and proposal choices unconstrained.At a population of 300, the allocation reaches 146–177 distinct topics.Source: §C.2 · p. 33
  • Tag extraction labels each 2025 paper's tasks and methods using a vocabulary that grows over publication order, followed by automated and manual merging of variants.The prompt requires substantive, discriminative tags, prohibits invention and results, and gives survey, review, and taxonomy papers no tags.Source: §C.2 · pp. 33, 35
  • LANDSCAPE gives agents a 2025 community reference of frequent task and method tags and task-method pairs, with their paper shares; it is a reference, not a menu.The block lists the 100 most common tags on each axis and 100 most common pairs, ordered by share.Source: §C.2 · pp. 36–37, Figure 14 · p. 36, Table 16 · p. 36

D LANGUAGE MODELS AND INFERENCE SETTINGS

  • The six language-model roles cover schema extraction, tag extraction, area screening, idea matching, and proposal writing, with model and inference settings specified for each.Schema extraction uses the same extractor and retry policy on both sides; the idea judge evaluates slot agreement, while all six proposal-writing models share one implementation, prompt, and 20-step budget.Source: D LANGUAGE MODELS AND INFERENCE SETTINGS · pp. 37–38

E ADDITIONAL ANALYSES

  • The additional analyses test repeat-run stability, model and exposure robustness, and report counts, schemas, and distributions that the main text summarizes.The section also identifies model pairings for its analyses and notes that two models are omitted from four analyses but included in other comparisons.Source: E ADDITIONAL ANALYSES · p. 38

E.1 MODEL SELECTION FOR EACH ANALYSIS

The additional analyses use DeepSeek-V4-Flash throughout, pairing it with other models according to each analysis's needs. The selected pairings broaden model coverage, while two models are omitted from these four analyses but included elsewhere.

E.1 MODEL SELECTION FOR EACH ANALYSIS

  • DeepSeek-V4-Flash is included in every analysis, with its partner selected for the analysis's needs and overall model coverage.Direction coverage uses DeepSeek-V4-Flash alone; slot decomposition pairs it with Llama 4 Scout, the highest-MATCH-IDEA model among the six.Source: §E.1 · p. 38
  • Case studies pair Qwen3.8-27B and DeepSeek-V4-Flash with examples chosen to show proposals that do and do not match a human paper's approach; scaling adds Gemini 3.5 Flash-Lite.The two case-study models provide the clearest triples meeting the example requirement, while Gemini broadens the scaling analysis beyond one model.Source: §E.1 · p. 39
  • Claude Haiku 4.5 and K2 Horizon 375B A23B are excluded from the four analyses but remain covered by the main table, human evaluation, contribution distribution, and leakage comparison.Those other comparisons report all six models.Source: §E.1 · p. 39

E.2 ROBUSTNESS AT A FIXED POPULATION SIZE

Repeat-run comparisons show small score changes in the tested settings. The benchmark fixes populations at 300 proposals while retaining the complete human publication set as the reference, enabling comparisons across models, methods, and areas.

E.2 ROBUSTNESS AT A FIXED POPULATION SIZE

  • Absolute changes across tested repeat runs are at most 0.44 percentage points for match metrics and 2.20 points for diversity metrics.The comparisons use two runs of 300 proposals in FREE and TOPIC settings for Gemini 3.5 Flash-Lite and DeepSeek-V4-Flash in world models and automated research.Source: §E.2 · p. 39

E.3 THE SIZE OF HUMAN REFERENCE

  • Table 2 scores each 300-proposal population against all 2026 papers in its area, whose counts range from 1,136 to 5,188, and measures the human level among those papers.The complete human reference represents what the community published and provides a common denominator for comparisons.Source: §E.3 · p. 39
  • Populations grow to 3.8–17 times the reference size in scaling analyses, where matching rises slowly and then nearly flattens, supporting 300 proposals as a cost-conscious choice.The authors keep the human reference complete and the LLM population at 300 because larger populations add inference cost without changing what IdeaTaste measures.Source: §E.3 · pp. 39–40

E.4 LEAKAGE AND BENCHMARK ROBUSTNESS

The authors test whether models released after 2025 benefit from exposure to 2026 papers and find that later releases do not lead on idea matching. Findings remain consistent across release-year groups, while the comparison with human researchers has an information-access boundary.

E.4 LEAKAGE AND BENCHMARK ROBUSTNESS

  • 3.10 on MATCH-IDEA for the 2025 controls exceeds 1.68 for the 2026 releases, while MATCH-CITE is 5.06 for 2026 releases against 4.14 for controls.Llama 4 Scout, with the earliest cutoff, scores highest on MATCH-IDEA; the authors conclude relevant-literature access is not what limits recovery.Source: §E.4 · p. 40
  • Across release-year groups, matching stays far below human levels: 2025 models reach 22% on citations and 26% on ideas, versus 28% and 14% for 2026 releases.No population in either group reaches human DIVERSITY-IDEA, and LANDSCAPE ranks last on MATCH-CITE in most settings of both groups.Source: §E.4 · p. 40
  • IdeaTaste can be repeated for any year by freezing the corpus at the prior year's end and comparing proposals with papers published in the target year.This setup permits evaluation against a reference published after a model's training data end.Source: §E.4 · p. 40
  • Human and LLM information access cannot be fully matched because authors' reading histories cannot be reconstructed after publication; the human score is therefore a reference, not a matched-condition arm.LLM agents retrieve only through 2025, while human authors may have accessed work released during 2026; LLM comparisons share the same corpus, budget, prompt, and judge.Source: §E.4 · p. 40

E.5 A POST-CUTOFF SLICE OF THE REFERENCE FOR LEAKAGE ANALYSIS

The post-cutoff split tests whether exposure to reference papers explains Gemini 3.5 Flash-Lite’s matching results. Citation matching does not improve with exposure, while idea matching is even when measured against each half’s human level.

E.5 A POST-CUTOFF SLICE OF THE REFERENCE FOR LEAKAGE ANALYSIS

  • Gemini 3.5 Flash-Lite is the only 2026 release with a stated March 2026 data cutoff, enabling a split between January–March and April-onward reference papers.The other three 2026 releases lack a stated cutoff or have release dates too late for a comparable split, so they are excluded.Source: §E.5 · pp. 40–41
  • Exposure does not help: citation matching is higher after the cutoff in all sixteen cells, while idea matching favors the earlier half in nine of sixteen cells against each half’s human level.Raw idea scores favor the earlier half in thirteen of sixteen cells, but human rows also favor it in three of four areas; the four methods rank identically across all three blocks.Source: §E.5 · p. 41, Table 19 · p. 41

E.6 CONTRIBUTION DISTRIBUTION OF LLM AGENTS

Across contribution types, overlap with earlier literature, and tag distributions, LLM populations depart from human research in both the forms their proposals take and how they combine tasks and methods. Their coverage of community directions does not translate into comparable approaches and plans.

E.6 CONTRIBUTION DISTRIBUTION OF LLM AGENTS

  • AI proposals concentrate more heavily on methods than human papers, while theory and position contributions are rare; Gemini 3.5 Flash-Lite writes methods in at least 99% of proposals in every area and arm.In automated research, DeepSeek-V4-Flash produces more artifacts than humans under FREE, TOPIC, and LANDSCAPE, and K2 Horizon 375B A23B does so under all four arms.Source: §E.6 · pp. 41–42, Figure 15 · p. 42

E.7 OVERLAP WITH ≤2025 LITERATURE

  • Every population repeats earlier literature less than 2026 papers do, but human papers restate earlier approaches and plans much more often; problems and motivations are closer.The human reference includes cross-year continuations by the same group, which raise its overlap, so lower proposal overlap does not establish greater originality.Source: §E.7 · p. 42, Table 20 · p. 43

E.8 TAG COVERAGE AND DISTRIBUTIONS

  • All populations concentrate more tag mass in their ten most-used tags than 2026 papers; even FRONTIER misses about half of human task–method-pair mass, while the other populations miss three fifths to three quarters.FRONTIER comes closest to human spread across tasks, methods, and pairs, but proposals carry more tags per document, so coverage partly reflects tagging more rather than spreading further.Source: §E.8 · pp. 43–44, Figure 16 · p. 43

E.9 CASE STUDY

These case studies illustrate how AI proposals can share human papers’ problems and motivations while differing in their approaches and research plans, or match across all four dimensions, with examples from KV-cache compression and code-benchmark contamination detection. Stored automatic slot judgments show these overlaps but are not human quality ratings, and the cases do not estimate how often the patterns occur.

Case 2: Online adaptation of world models

  • Stored labels mark ERALD as matching problem and motivation but not approach or research plan, while DynaLoRA matches all four slots.These are automatic agreement judgments, not human quality ratings.Source: §E.9 · pp. 44, 48, 49

Case 1: Expert reuse in memory-constrained inference

  • In Case 1, TopicRoute predicts expert demand from prompt topics while ReMoE and LARC fine-tune routing; labels mark TopicRoute as matching only problem and motivation, and LARC as matching all four slots.The contrast is between anticipating expert demand and changing routing behavior.Source: §E.9 · pp. 44, 46

Case 2: Online adaptation of world models

  • In Case 2, SPREAD and DynaLoRA adapt frozen world models with low-rank adapters, whereas ERALD retrieves episodic transitions and fuses them with a frozen model; DynaLoRA’s adapter composition is an additional extension, not the shared mechanism.ERALD absorbs regime shifts by storing observed transitions without updating transition-model weights.Source: §E.9 · pp. 47–48

Case 3: Joint KV-cache eviction and quantization

  • In Case 3, RDKV and ASYM-RD use rate-distortion and reverse water-filling, while TriAdmit learns retention scores for a three-action controller; the shared principle is joint storage allocation, not identical implementations.RDKV and ASYM-RD also differ in distortion definitions and allocation timing.Source: §E.9 · p. 49

KV-cache compression

  • RDKV outperforms the best evaluated baseline by 9.1% on average, recovers 97.81% of full-cache accuracy with 2.48% cache retention, and achieves 4.5x decode speedup and 1.9x peak memory reduction at 128K context.The speed and memory comparison is against full-cache FlashAttention-2 decoding, with comparable performance maintained.Source: §E.9 · p. 49
  • TriAdmit matches the problem and motivation but not the proposed approach or research plan in the automatic judgments.It learns per-(token, layer, head) retain, demote, or evict actions and uses an online solver for byte and attention budgets; its plan tests whether three actions beat two-action oracles and whether a distilled scorer approaches the oracle.Source: §E.9 · p. 50
  • ASYM-RD matches the problem, motivation, proposed approach, and research plan in the automatic judgments.It allocates bits jointly using attention-output distortion and reverse water-filling, with tests against compression baselines at equal KV bytes and a comparison of predicted key sensitivity with attention scores.Source: §E.9 · p. 51

Code-benchmark contamination detection

  • SrDetection improves average F1 over strong baselines by 21.52 points in gray-box evaluation and 14.46 points in black-box evaluation.The framework generates semantically equivalent benchmark variants and flags cases where the original is disproportionately easier; its evaluation also studies 15 Code LLMs on four benchmarks.Source: §E.9 · pp. 51–52
  • Shrinking Solution Ensembles matches SrDetection’s problem and motivation but not its proposed approach or research plan in the automatic judgments.The proposal uses controlled contamination and tests whether leaked problems yield less diverse correct solution ensembles, measured by cluster entropy and self-overlap, than matched unseen problems.Source: §E.9 · p. 52
  • ReformGap matches the problem, motivation, proposed approach, and research plan in the automatic judgments.It detects possible memorization through the difference between pass@k on an original code problem and execution-verified equivalent reformulations, testing under simulated contamination and on API-only models.Source: §E.9 · p. 53

Related papers

Loading 2611.03424v1…