PaperlayerView exact revision on arXiv

Source-linked AI summary

RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases

Yingqian Wu, Jingcong Liang, Siyuan Wang, Zhenfei Yin, Philip Torr, Junchi Yu, Zhongyu Wei

arXiv:2609.10092v1cs.AIcs.CL

TL;DR

LLM research agents need evaluation methods for forecasting how research attention will shift, not only for summarising existing literature. RAP provides a rolling benchmark of temporally restricted forecasts across fixed research directions and finds that retrieval helps selectively but agents generally remain below EWMA, with cumulative-history acquisition and beyond-persistence updating as linked bottlenecks.

  • Problem

    Existing evaluations do not test repeated agentic forecasting of a jointly normalised activity distribution over the same frozen within-field directions.

  • Method

    RAP evaluates six-month paper-share forecasts across eight frozen directions in 278 fields and 1,390 episodes under temporally restricted evidence access.

  • Results

    Search generally helps, but all four diagnostic models remain below EWMA; State carry-forward beats Forecast under cumulative history, while exact history yields only a slight GPT-5.5 advantage over EWMA.

  • Takeaways & Limitations

    Forecast-oriented targets can redirect evidence acquisition toward less recent information, and improving research-attention forecasting requires separating evidence selection from future-specific updating.

  • Takeaways & Limitations

    ArXiv submission shares proxy research activity rather than scientific quality, impact, or novelty, and the eight directions are operational coordinates rather than a unique expert taxonomy.

Abstract

from arXiv · show

Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate because reviews and research ideas lack uniquely verifiable outcomes. We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes. At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions. Search generally helps, but all four diagnostic models perform worse than an exact-count exponentially weighted moving average (EWMA) baseline in compositional accuracy. We identify two linked bottlenecks. Under cumulative-history access, State carry-forward outperforms direct Forecast for all four diagnostic models; frozen-evidence replay links a shared component of this reversal to Forecast-oriented policies retrieving a smaller share of recent evidence. Even with exact historical activity, future-specific updating remains limited, with only GPT-5.5 plus reopened Search slightly surpassing EWMA. Fine-tuning on realised outcomes improves Qwen3-4B's forecast Spearman correlation by 0.105 on held-out fields at later origins, with gains also on change-rich episodes.

1 Introduction

RAP addresses the gap between retrospective literature synthesis and forecasting future research attention by evaluating LLM agents on rolling, outcome-grounded activity distributions. The benchmark reveals that retrieval helps selectively, but agents remain below EWMA and exhibit evidence-acquisition and updating bottlenecks.

  • Motivation: RAP evaluates whether LLM agents can forecast future research attention rather than merely summarise published literature.It targets the distribution of scientific activity across a fixed field-specific direction slate.
  • Benchmark: 1,390 episodes across 278 AI/ML fields are constructed from temporally grounded rolling forecasts of six-month paper distributions.Historical evidence and future outcomes use first-submission dates, enabling comparable evaluation across origins.
  • Benchmark: RAP measures research attention as the relative distribution of papers across directions, not scientific importance, novelty, or value.The protocol compares Closed, Fixed-window, and Expanding-history access, with State carry-forward as a persistence control.
  • Findings: Search generally improves over Closed, but no natural agent condition surpasses the exact-count EWMA baseline in compositional accuracy.Pre-cut-off activity nevertheless contains measurable signal about future departures from persistence.
  • Findings: Under cumulative-history access, Forecast-oriented policies retrieve less recent evidence, while State carry-forward outperforms direct Forecast for all four diagnostic models.Outcome-aligned fine-tuning improves Qwen3-4B by +0.105 on later-origin episodes without establishing a mechanism-level repair.

2 Related Work

Prior work covers literature synthesis, scientific forecasting, temporal evaluation, and research-agent workflows, but RAP targets a distinct outcome: repeated forecasting of joint research-activity distributions over fixed within-field directions.

  • Scientific forecasting: Earlier scientific forecasting studies model topic evolution, field activity, topic prevalence, future links, high-impact concepts, experimental results, and scientific events.These works span aggregate trends and individual scientific artefacts.
  • Temporal evaluation: Temporal-evaluation research motivates rolling, dynamically constructed tests because reported cut-offs may differ from effective knowledge horizons.RAP follows rolling-origin practice and compares persistence, exponential smoothing, and no-change references.
  • Research agents: Research-agent systems support literature-grounded discovery, retrieval-based synthesis, automated surveys, idea generation, execution, drafting, and review.These evaluations often emphasise final artefacts rather than controlled evidence-use comparisons.
  • RAP’s distinction: RAP holds the field, cut-off, evidence universe, interface, and codebook fixed while varying State versus Forecast to separate evidence acquisition from terminal readout.This positions RAP as a controlled benchmark for agentic forecasting rather than literature summarisation.

3 Research Attention Prediction

RAP combines frozen field-specific measurement coordinates with rolling, temporally restricted agent forecasts of eight-way research-attention compositions. Its validity checks establish reliable persistence, measurable departures, and robustness to counting ambiguity.

  • Construction: RAP combines a frozen measurement instrument with a rolling search-and-Forecast protocol over field-specific corpora and outcome labels.The instrument uses overlapping corpora, pre-2024 codebooks, and a date-blind assignment rule.
  • Construction: Exactly eight operational directions are consolidated per field from pre-2024 evidence and frozen across rolling origins.Each direction has definitions, inclusion and exclusion boundaries, and exemplars.
  • Targets: The realised target is an eight-dimensional composition of direction shares over each successive six-month future window, excluding other from normalisation.The eight directions cover 0.994 of future-window memberships on average.
  • Evaluation: Agents adaptively query temporally eligible papers and return eight non-negative percentage weights summing to 100 under Closed, Fixed-window, or Expanding-history access.Forecasts are scored by episode-level Spearman agreement, with magnitude-sensitive checks from TV and JSD.
  • Validity: 0.803 is the Test EWMA score, while a 141-episode change-rich subset yields EWMA 0.744 and residual-direction alignment +0.151 [+0.013, +0.288].These results indicate strong persistence alongside measurable, predictable departures.
  • Validity: Boundary-aware rescoring preserves the five-condition ordering in 15 of 16 model–policy combinations.The sole exception is a 0.002 near-tie for GPT-OSS-120B, while all models retain the key policy-effect signs.

4 Main Results

Retrieval improves forecasting selectively, but cumulative history does not improve Forecast over a recent fixed window and agents remain below persistence baselines. The results separate recovering current activity levels from reliably updating beyond persistence.

  • Baseline comparison: Across all origins, Recent persistence, EWMA, and Dev-tuned ARIMA outperform the strongest agent condition.EWMA remains the primary persistence anchor because it is transparent, competitive, and compositional.
  • Retrieval performance: Fixed-window retrieval improves over Closed for six of seven agents, whereas Expanding-history is lower than Fixed-window for all seven.The largest cumulative-history drop occurs for GPT-OSS-120B; DeepSeek-V3 is essentially unchanged.
  • Beyond persistence: No agent condition improves TV over EWMA across GPT-5.5, Qwen3.6-27B, GPT-OSS-120B, and DeepSeek-V3.Agents recover useful activity levels without reliably converting retrieval into calibrated updating beyond persistence.
  • Strict evaluation: At the shared Forecast-Strict origin, Fixed-window remains above Closed and Expanding-history remains below Fixed-window for the eligible models.All eligible agents remain below EWMA and ARIMA.
  • Interpretation: The cumulative-history failure motivates separating evidence acquisition from terminal readout rather than attributing every forecasting failure to acquisition.The next diagnostic stage tests whether the reversal is carried by retrieved evidence or by how identical evidence is used.

5 Diagnosing Target-Conditioned Search Failure

The diagnosis separates evidence acquisition from terminal forecasting: under cumulative-history access, State carry-forward beats direct Forecast, and replay attributes part of the reversal to what evidence is retrieved. Exact-history interventions then show that future-specific updating remains limited.

  • 5 Diagnosing Target-Conditioned Search Failure: State-as-Forecast outperforms Forecast for all four diagnostic models under Expanding-history, while Fixed-window gaps are smaller and sign-inconsistent.The reversal persists in every eligible post-cut-off comparison and is specific to selecting time-relevant evidence from a long record.
  • 5 Diagnosing Target-Conditioned Search Failure: Frozen replay shows that State-oriented evidence improves Forecast readouts under a common objective, making acquisition a shared component of the reversal.After matching Search-call budgets, the effect is robust for GPT-5.5, GPT-OSS-120B, and DeepSeek-V3, but positive and uncertain for Qwen3.6-27B.
  • 5 Diagnosing Target-Conditioned Search Failure: State Search places 100% of returned paper slots in the preceding six months, versus 47–80% for Forecast.State produces fewer distinct papers for three models but more for Qwen3.6-27B, and its evidence better matches recent and future realised distributions for every model.
  • 5 Diagnosing Target-Conditioned Search Failure: Forecast-specific recent-window options improve evidence–future alignment by +0.086– +0.124, whereas the State-query advantage is at most +0.011 and negative for three models.The counterfactual identifies temporal-scope allocation as a shared Search signature, not a general advantage of State query wording.
  • 5 Diagnosing Target-Conditioned Search Failure: With exact historical distributions, every model shows positive residual-direction alignment, but only GPT-5.5 with reopened Search surpasses EWMA.No model converts within-origin residual signal into positive six-month revision tracking.
  • 5 Diagnosing Target-Conditioned Search Failure: Exact-history interventions expose two boundaries: recovering time-local state is difficult, and explicit Forecast must add a reliable update beyond persistence.GPT-5.5 recovers most of the level gap with exact history, but exact history alone yields no reliable advantage over EWMA.

6 Discussion and Limitations

RAP identifies state recovery and beyond-persistence updating as distinct empirical boundaries, while showing that target-conditioned retrieval can redirect evidence acquisition. Its conclusions are bounded by the operational arXiv-share construct, retrospective sampling choices, and temporal-exposure controls.

  • Discussion: RAP separates recovering a useful time-local activity level from adding a reliable future update beyond persistence.Exact historical activity lets GPT-5.5 recover most of the level gap, but only GPT-5.5 with reopened Search slightly surpasses EWMA.
  • Discussion: The Forecast target can change queries, retrieved papers, and search termination, so final-answer evaluation can misattribute acquisition failures to synthesis.State-oriented trajectories support better Forecast readouts even when Search-call counts are matched.
  • Limitations: All-Origin analyses may include episodes within a model’s knowledge horizon, so post-cut-off forecasting claims are reserved for Forecast-Strict cells.Exposure-stratified rescoring preserves the main Expanding-history and State–Forecast directions in eligible strict comparisons.
  • Limitations: RAP measures next-window arXiv submission shares rather than scientific quality, impact, novelty, or value.The eight directions are operational coordinates for reallocation among established programs, not a unique expert taxonomy.
  • Limitations: The field universe and membership instrument use retrospective information through the end of 2025, although model-facing retrieval remains cutoff-restricted.The field universe, direction slates, assignments, and dependency-aware splits were frozen before Test outputs were inspected.
  • Data and construction: The frozen source snapshot contains 356,357 papers from four arXiv categories, and the pipeline constructs overlapping field corpora with model-assisted extraction and expansion.Retaining fields with at least 300 memberships yields 281 corpora, of which 278 admit valid direction codebooks.

A.3 Retrospective field-universe conditioning

Retrospective field-universe conditioning changes the sampling frame but leaves the central comparisons broadly intact. The analysis also documents that the direction codebook is an operational, model-assisted measurement instrument rather than a uniquely validated taxonomy.

  • A.3 Retrospective field-universe conditioning: Across 21 model–condition cells, restricting to pre-2024-eligible fields changes estimates by at most .046.Near-tied condition orderings can reverse, so the sensitivity supports central comparisons without establishing prospective-sampling equivalence.
  • A.3 Retrospective field-universe conditioning: The Expanding-history State-as-Forecast advantage remains positive on the restricted subset: .016, .072, .057, and .048 for Qwen3.6-27B, GPT-OSS-120B, DeepSeek-V3, and GPT-5.5, respectively.These values preserve the direction of the central comparison despite the retrospectively conditioned field universe.
  • Direction codebook: The eight directions are operational measurement coordinates, neither a unique natural taxonomy nor necessarily exhaustive or mutually exclusive.Their construction uses pre-2024 evidence and a frozen exact-eight consolidation.
  • Direction codebook: 55.48% of paper–field memberships receive a single-direction label, while 42.60% are boundary ties and 1.58% are genuine bridges.The canonical primary label is therefore a counting convention rather than a noiseless or unique paper-level gold label.
  • Direction codebook: Model-assisted construction includes a Claude Sonnet 4.6 teacher, a distilled Qwen3.5-4B extractor, GPT-5.5 codebook contributions, and Qwen3.6-27B paper assignment.An independently drafted Opus instrument produces similar persistence scores and target repeatability on 38 fields under cross-fit alignment.

B.1 Human operational-assignability audit

The audit tests whether a written direction codebook and paper metadata support reproducible operational assignment. A single label-blind annotator generally recovered corpus field membership, but the study is not an inter-annotator or expert-validation study.

  • Audit design: The sample covered 24 construction-stratified fields and included single-direction, boundary/bridge, population-positive, and retrieved non-primary candidates.The annotator recorded field membership, primary and optional secondary directions, clarity, and confidence.
  • Audit findings: 0.958 [0.917, 0.992] of frozen-positive pairs reproduced field membership, and directions were broadly distinguishable in 23 of 24 fields.The audit used titles, abstracts, operational field definitions, and eight written direction definitions.
  • Interpretation: Only 5.2% of accepted items received a secondary label, so exact primary match is a lower bound on conformance rather than an accuracy estimate.For two-direction candidates, 40 of 58 choices fell inside the declared pair; single-direction candidates had 34 exact matches among 62 items.
  • Interpretation: The Forecast-target reweighting renormalises cell shares over 2024+ memberships because pre-2024 papers cannot enter a Forecast target window.The correction materially lowers apparent conformance.
  • Limitations: The audit evaluates operational assignability with one label-blind annotator, not inter-annotator agreement or field-expert validation.Its conclusions therefore concern usability of the written instrument rather than population-wide annotation validity.

B.2 Robustness to assignment ambiguity

Robustness checks show that assignment ambiguity changes absolute scores but preserves the paper’s main structural comparisons. The evidence also bounds interpretation because the underlying instrument uses a counting convention and several evaluation settings are explicitly controlled or retrospective.

  • Ambiguity robustness: 0.123 total-variation distance between paired labellers’ pooled marginals was consistent with the exchangeable-disagreement null (p = 0.21), with no slot showing net flow excluding zero.This is a non-detection, not proof of unbiasedness; the null’s 95th percentile is 0.149.
  • Ambiguity robustness: Complete five-condition ordering was preserved in 15 of 16 model–policy combinations, with only a 0.002 GPT-OSS-120B near-tie as an exception.Every model retained the signs of A1–A0, A3–A1, and A4–A3 under every counting policy.
  • Instrument stability: The second assignment instrument showed stable agreement across the 2024 boundary, supporting temporal robustness alongside operational assignability.The cited passage reports a paired change of −0.009 before continuing beyond the supplied excerpt.
  • Temporal scope: Forecast-Strict cells require the complete future window to follow the provider-reported or conservatively bounded checkpoint cutoff.All-Origin results are interpreted as controlled retrospective evidence use rather than necessarily unseen forecasting.

D.2 Compositional-distance sensitivity

The analysis evaluates compositional forecasting with total variation and Jensen–Shannon distance, alongside persistence-controlled residual measures. State carry-forward improves ordinal agreement across all four models, but complete compositional gains over EWMA remain absent in natural-evidence conditions.

  • Metrics: Lower total-variation and Jensen–Shannon distances indicate better agreement between predicted and realised eight-direction compositions.The percentage interface exposes relative tradeoffs and share magnitude; both distances are computed on normalized vectors.
  • State comparison: State-as-Forecast improves Spearman agreement over explicit Forecast for all four diagnostic models under Expanding history.The gains are +0.041 for GPT-5.5, +0.035 for Qwen3.6-27B, +0.082 for GPT-OSS-120B, and +0.051 for DeepSeek-V3.
  • State comparison: Share magnitude improves for GPT-5.5, Qwen3.6-27B, and DeepSeek-V3, while GPT-OSS-120B improves ordinal ranking but worsens TV and JSD.The corresponding TV/JSD changes are +0.010/+0.003, +0.006/+0.002, and +0.007/+0.003; GPT-OSS-120B changes by −0.006/−0.003.
  • Persistence control: EWMA reaches 0.803 on Test against a reliability-implied ceiling of 0.911, leaving an absolute gap of 0.108.The primary estimand is next-window level composition, for which persistence is valid predictive information.
  • Persistence control: Every natural-evidence A0/A1/A3 agent cell has negative TV gain relative to EWMA, while residual cosine is weak or absent across models.At the latest origin, residual cosine is at most 0.023 and unresolved or negative in every cell.
  • Stress tests: The change-rich subset contains 141 of 1,055 Test episodes and is selected using an episode-specific stationary null with p ≤0.05 and repeatability at least 0.5.Because selection reduces persistence partly by definition, claims on this subset rely on baseline-controlled or paired intervention metrics.

E.6 Robustness scope

Robustness analyses preserve the main structural condition contrasts under alternative assignment policies and retrieval audits, but they do not establish invariance of the full leaderboard or unbiased prevalence estimates.

  • Assignment robustness: Assignment-policy rescoring preserves all three structural condition contrasts for all four diagnostic models.The restricted-field analysis also preserves the State-as-Forecast advantage, with only near-tied retrieval contrasts reversing for two models.
  • Assignment robustness: Alternative assignment policies do not support invariant seven-model leaderboard ordering across every direction slate or policy.One exact five-condition ordering has a near-tied GPT-OSS-120B exception under the single-direction-only target.
  • Retrieval audit: Search-trace analysis treats retrieved direction composition as a behavioral diagnostic rather than an unbiased estimator of field prevalence.Active queries condition which papers are returned, so the returned set is not interpreted as a direct prevalence sample.
  • Retrieval audit: Re-executing 86,284 frozen queries under common recent-window options reduces the native State-query evidence–future advantage to at most +0.011 and makes it negative for three models.Native calls reproduce recorded paper IDs and ordering exactly.

F.3 Oracle historical-state ablation

The oracle historical-state ablation separates evidence access from future updating. Exact history narrows the level gap to persistence and yields partial departure alignment, but reopening Search only modestly improves the complete Forecast and the intervention is not deployable.

  • Oracle intervention: Exact history closes most of the level gap between natural Search and persistence, but Counts-only does not robustly surpass EWMA as a complete Forecast.The ablation supplies exact pre-cutoff direction counts, bypassing evidence acquisition and semantic aggregation.
  • Oracle intervention: Counts-only achieves corrected residual Spearman of +0.163 [+0.108, +0.220], indicating partial alignment with departures from EWMA.Reopening Search improves over Counts-only by +0.006 [+0.003, +0.009].
  • Replay: Full-evidence replay generates independent Forecast and State readouts from byte-identical serialized Search evidence.The replay excludes source prompts, hidden reasoning, terminal answers, and rationales.
  • Adaptation: Full-SFT raises Forecast scores from 0.310 to 0.382 with Fixed-window access and from 0.227 to 0.364 with Expanding-history access.The pooled gain remains resolved on 63 model-output-independent change-rich episodes.
  • Adaptation: Full-SFT improves the end-to-end RAP policy but does not isolate repaired temporal-updating or evidence-acquisition mechanisms.The comparison is neither Search-call-matched nor compute-normalised.
  • Scope: The Qwen3-4B adaptation establishes cross-field learning on unseen post-release outcomes, not learning from genuinely post-cutoff supervision or transfer to open-ended literature synthesis.The checkpoint was released in April 2025, but its exact earlier knowledge cutoff is undocumented.

G.3 Limitations and claim boundaries

RAP’s claims are bounded by its operational measurement choices, temporal eligibility rules, and evaluation protocols. Absolute scores depend on fixed assignment conventions, while some experiments support narrower conclusions than the full leaderboard.

  • Measurement scope: RAP measures directional arXiv submission activity, not scientific quality, impact, novelty, breakthrough probability, or complete literature-review quality.Fields may overlap, and each eight-direction slate is a fixed candidate coordinate system rather than an exhaustive or uniquely natural partition.
  • Temporal scope: Field eligibility is retrospectively conditioned on corpus growth through 2025, although codebooks use pre-2024 evidence and episodes expose only their registered pre-T Search universe.
  • Measurement scope: Paper-level boundary assignments make absolute scores dependent on a frozen counting convention.The alternative Opus instrument supports robustness of target repeatability and persistence-level conclusions only on its 38-field subset, without rerunning the seven-agent leaderboard.
  • Validation scope: The human audit supports written-codebook usability but does not establish inter-annotator agreement, domain-expert endorsement, or uniqueness of the primary label.It is a single-annotator, label-blind operational-assignability audit.
  • Inference boundaries: Only Forecast-Strict cells support unseen-future claims, while the SFT experiment establishes within-RAP adaptation to unseen later outcomes under a potentially undocumented and pretraining-contaminated cutoff.The SFT outcomes are post-release and dependency-disjoint for the frozen Qwen3-4B checkpoint, but earlier knowledge-cutoff status is undocumented.
Loading 2609.10092v1…