PaperlayerView exact revision on arXiv

Source-linked AI summary

Positioning manuscripts in the scientific landscape with agentic AI

Jiawen Chen, Zichen Zhang, Bingxuan Li, Quan Sun, Yiyan Zhang, Edric Tam, Jinjie Lin, Didong Li, Yun Li, Bingxin Zhao

arXiv:2609.13760v1cs.AI

TL;DR

Choosing a suitable journal is time-consuming and uncertain, and existing AI support for publication decisions is limited. PASS combines manuscript understanding with literature retrieval and journal-fit reasoning. It achieved 50.3% Top-1 accuracy and 86.1% Top-5 accuracy on a leakage-audited benchmark, outperforming LLM baselines and established journal-selection tools.

  • Problem

    Journal selection requires judging fit across manuscript contribution, journal scope, audience, and novelty expectations, but remains time-consuming and uncertain.

    Source: §1 · pp. 1–2

  • Method

    PASS structures a manuscript’s research question, contribution, claims, and limitations, retrieves domain-specific literature, and reasons over field-specific journal spaces to rank venues.

    Source: §1 · p. 2, §2.1 · p. 3

  • Results

    50.3% Top-1 accuracy and 86.1% Top-5 accuracy on 2,184 preprints across 16 biomedical fields exceeded state-of-the-art LLM baselines and established journal-selection tools.

    Source: §1 · p. 2

  • Takeaways & Limitations

    Journal landing can be treated as a measurable endpoint of publication-oriented manuscript understanding, with literature context essential to reliable prediction.

    Source: §3 · p. 13

    The benchmark records only each manuscript’s final publication journal, not its submission trajectory, rejection stage, or revision history.

    Source: §3 · p. 13

Abstract

from arXiv · show

Publishing a research manuscript is a routine yet demanding part of scientific life: time-consuming, stressful, and often uncertain in outcome. Recent advances in large language model (LLM)-based agentic AI have shown promise across a range of scientific tasks, and here we ask whether agentic AI can help researchers navigate the publication process itself by reliably inferring a manuscript's eventual publication venue from its content and literature context. We introduce PASS (Publication-oriented Agentic Scientific System), an agentic system that understands manuscripts within their domain-specific literature context and predicts top-matched publication venues. PASS positions each manuscript within its surrounding literature landscape by reconstructing its local scientific neighborhood, tracing its topic trajectory, and reasoning over field-specific journal spaces. Evaluated on a leakage-audited benchmark of over 2,000 preprints across 16 biomedical fields, PASS achieved Top-1 accuracy of 50.3% and Top-5 accuracy of 86.1%, outperforming state-of-the-art LLM baselines and established journal-selection tools. PASS-produced quality scores, such as impact potential and novelty, aligned with independent measures of publication outcome. We also found that the designed literature retrieval module is the strongest performance contributor, particularly for positioning manuscripts relative to nearby work, and that PASS maintained near-full performance from the abstract alone, whereas LLM baselines required the full manuscript text. An independent human evaluation found strong researcher agreement with PASS's manuscript understanding and recommendation rationale. PASS has been released as a public platform (https://ratemypaper.ai/) for broad researcher access.

1 Introduction

Choosing a journal requires judging a manuscript’s fit to scope, audience, novelty expectations, and related literature, and the process is time-consuming and uncertain. PASS addresses this underexplored publication-stage task by reasoning from manuscript content and literature context to recommend journals.

1 Introduction

  • Journal selection is uncertain and consequential, while earlier AI publication work focused mainly on intermediate research outputs and existing tools on short-text similarity or abstract classification.A poor journal match may delay publication and increase revision and resubmission burdens.Source: §1 · p. 2
  • 50.3% Top-1 accuracy and 86.1% Top-5 accuracy on 2,184 preprints across 16 biomedical fields exceeded state-of-the-art LLM baselines and established journal-selection tools.The benchmark was temporally controlled and leakage-audited.Source: §1 · p. 2
  • Literature retrieval was PASS’s strongest performance contributor, and PASS retained near-full accuracy from abstracts while LLM baselines relied more heavily on full manuscripts.The abstract, introduction, and discussion frame a manuscript’s contribution and field position.Source: §1 · pp. 2–3
  • PASS’s manuscript-level quality assessments were associated with independent measures of publication outcome.The assessments included novelty, impact, conceptual soundness, and writing quality.Source: §1 · p. 2
  • On the public platform, 88.0% of recommended journal tiers were judged close to expectations, and researchers agreed with 88.7% of manuscript-level assessments.The independent evaluation included six doctoral-level reviewers across 44 reports.Source: §1 · p. 3

2 Results

PASS combines manuscript understanding, literature retrieval, and journal-fit reasoning to predict publication venues and produce evidence-grounded assessments. Across the evaluated fields, it outperformed GPT-5 baselines, its scores tracked journal tier, and researchers rated its reports and recommendations favorably.

2.1 PASS: an agentic framework for literature-grounded journal recommendation

  • PASS uses three coordinated agents to structure manuscript contributions, position them against retrieved literature, and reason about journal compatibility.The workflow combines manuscript-derived signals with closely related studies and broader review or benchmark context, then returns ranked journals, rationales, and manuscript-level assessments.Source: §2.1 · pp. 3–4, Figure 1 · p. 4

2.2 PASS improves journal-landing prediction across biomedical fields

  • PASS achieved 50.3% Top-1 and 86.1% Top-5 accuracy, leading task-engineered GPT-5 by 9.1 and 7.0 percentage points across 16 evaluation panels.It had the highest Top-K accuracy in every evaluated panel and outperformed JANE across all 16; on Pubmender’s test split, it exceeded Pubmender by over 30 percentage points at both Top-1 and Top-5.Source: §2.2 · pp. 5–6, Figure 2 · p. 6

2.3 PASS scores are interpretable and track journal tier

  • Across 2,184 manuscripts, mean overall PASS scores rose with journal visibility and selectivity; the highest versus lowest SJR bins differed by 1.04 points.Dimension scores were also associated with more selective journals: impact potential had the largest SJR-bin separation (1.41 points), while novelty had the largest H-index-bin separation (1.36 points).Source: §2.3 · p. 7, Figure 3 · p. 8

2.4 Ablation identifies drivers of journal-landing prediction

  • Removing literature evidence erased the largest share of PASS’s gain over task-engineered GPT-5 (29.2%), versus manuscript signals (22.5%) and task instructions (10.1%).Removing related studies alone erased 14.6% of the gain versus 7.3% for reviews or benchmarks; retrieval contributed more to prediction for discovery and method papers than resource papers.Source: §2.4 · pp. 7, 9, 10, Figure 4 · p. 9

2.5 Agentic reasoning preserves journal-landing signal in concise sections

  • With abstract-only input, PASS achieved 53.8% Top-1 and 85.9% Top-5 accuracy, close to full-manuscript results of 53.0% and 86.8%.Abstract-only input matched 101% of full-text mean Top-K performance; task-engineered and naive GPT-5 reached 94% and 91%, respectively, using the abstract, and remained below their full-text benchmarks across section subsets.Source: Figure 5 · p. 10, §2.5 · pp. 10–11

2.6 Human evaluation supports the accuracy and usefulness of PASS reports

  • Across 44 human-evaluated reports, researchers agreed with PASS’s manuscript understanding in all cases (100%) and with topic identification in 40 of 44 (91%).Among 43 tier ratings, 88% were close to expectations; researchers found roadmaps helpful in 41 of 44 reports (93%), with 80.2% of comments useful on average. Agreement scores exceeded 0.8 for overall assessments and strengths across all four dimensions, while weakness ratings remained positive with a minimum mean of 0.58.Source: §2.6 · pp. 11–12, Figure 6 · p. 12

3 Discussion

PASS treats journal landing as a measurable task of publication-oriented manuscript understanding, combining manuscript analysis, literature retrieval, and journal-fit reasoning. The discussion highlights literature context and concise contribution-framing sections as key signals, while noting that the benchmark records only final publication journals.

3 Discussion

  • PASS integrated manuscript understanding, literature retrieval, and journal-fit reasoning to produce interpretable recommendations and feedback, outperforming GPT-5 baselines and established journal-recommendation tools.The authors frame journal landing as evidence-grounded reasoning over a manuscript and its scientific context, rather than simple topical matching.Source: §3 · p. 13
  • Literature retrieval had the largest ablation effect, with related studies informing novelty and competitive positioning while reviews and benchmarks capture broader field expectations.The authors describe these sources as complementary inputs for reconstructing a manuscript’s scientific neighborhood.Source: §3 · p. 13
  • PASS recovered near-full journal-landing performance from abstracts, introductions, and discussions, whereas GPT-5 baselines needed full text for their best performance.The authors suggest this can support publication-oriented feedback before a manuscript is complete, including from abstract-only input.Source: §3 · p. 13
  • The public platform presents its manuscript understanding, retrieved evidence, recommendation rationales, and improvement roadmap as decision support rather than a substitute for editorial judgment.The authors characterize its value as making journal-fit reasoning transparent, inspectable, and actionable.Source: §3 · p. 13
  • The benchmark observes only each manuscript’s final publication journal, not its submission sequence, rejection stages, or revisions.The authors identify prospective submission-trajectory data as a future resource for more adaptive and personalized guidance.Source: §3 · p. 13
  • The authors envision agentic systems helping researchers monitor domain trends, identify missing comparisons, benchmark against field standards, and refine study framing while work is developing.They state the goal is appropriate audience selection and reduced avoidable mismatch, not prestige optimization.Source: §3 · p. 14

4 Online Methods

The methods describe PASS’s agentic workflow, comparison systems, benchmark construction and leakage audit, and journal-metric sources. They also specify the benchmark and platform’s public availability.

4.1 PASS agentic system details

  • PASS parses manuscript context into a structured profile, retrieves date-eligible related studies and reviews or benchmarks, then uses that evidence for journal-fit reasoning.The evaluation context included the title, abstract, introduction, and discussion; retrieval records were ranked and divided into manuscript-adjacent and broader field evidence.Source: §4.1 · pp. 16–17
  • Three GPT-5 runs combine manuscript and literature evidence with author context as a secondary tie-breaker, then fuse ranked lists into five recommendations with rationales and feedback.The journal-fit agent assesses quality and compatibility, while probability-weighted reciprocal-rank fusion aggregates the independent runs.Source: §4.1 · p. 17

4.2 Naive and task-engineered GPT-5 baselines

  • Both GPT-5 baselines used the same evaluation panels and labels without retrieval; the task-engineered version added a rubric and journal metrics, while the naive version used minimal instructions.Both were queried three times and used the same aggregation procedure as PASS.Source: §4.2 · pp. 17–18

4.3 Benchmark dataset construction

  • The final benchmark comprised 2,184 published preprints across 16 panels, with candidate journals selected by mapped SCImago subject area and eventual publication journal.The cohort was filtered for verified publication links; categories with more than 200 eligible manuscripts were subsampled to 200.Source: §4.3 · p. 18

4.4 Benchmark data leakage audit

  • In a GPT-5 recall audit of all 2,184 benchmark papers, the model reported knowing none of their destination journals.The audit prompt included title, authors, and abstract and constrained the response to yes or no.Source: §4.4 · p. 18

4.5 Comparison with established journal-selection systems

  • JANE’s top-30 outputs were filtered for preprint repositories and likely self-matches; Pubmender was reimplemented and trained on a benchmark subset, with PASS and JANE evaluated on the same 228-paper set.Pubmender used an 8:1:1 training, validation, and test split; its best-validation-Top-5 checkpoint was selected.Source: §4.5 · p. 19

4.6 Journal-level metrics

  • A fixed 2024 SCImago snapshot supplied SJR for journal ranking, two-year citations for short-window impact, and journal H-index for cumulative impact.The same snapshot was used for candidate-journal construction, ranking, and downstream analyses.Source: §4.6 · p. 19

Code and data availability

  • The PASS platform and benchmark metadata are publicly available online.The metadata include preprint identifiers, publication labels and dates, candidate-journal sets, and SCImago-derived journal metrics.Source: Code and data availability · p. 19
Loading 2609.13760v1…