PaperlayerView exact revision on arXiv

Source-linked AI summary

PaperScorer: How Predictable Is Paper Acceptance?

Sachin Konan, Jonathan Liu, Zhuang Liu

arXiv:2610.31791v1cs.LGcs.CV

TL;DR

Peer review is noisy, and benchmark scores do not capture every research contribution, raising the question of how much the paper alone predicts its decision. PaperScorer predicts accept/reject outcomes from anonymized paper text or screenshots. Vision models outperform text and frontier baselines, with signal concentrated at rating extremes and calibrated confidence; the score is not a measure of novelty or long-term impact.

  • Problem

    Peer review is noisy and depends on reviewer assignment, interpretation, and aggregation, so the paper asks how predictable final decisions are from paper content alone.

    Source: §1 · p. 2

  • Method

    PaperScorer trains text and vision models on final accept/reject outcomes from anonymized papers, removing author identity and formatting shortcuts.

    Source: §1 · p. 2, §3.2 · p. 6

  • Results

    Vision models outperform text and frontier systems; PaperScorer-Vision reaches 68% accuracy on ICLR, and its accuracy is strongest at low and high reviewer ratings while intermediate-rated papers remain near chance.

    Source: §1 · p. 2

  • Takeaways & Limitations

    PaperScorer provides a calibrated estimate of acceptance probability for use as a prior in workflows such as agentic reviewing and literature search, not a replacement for peer review.

    Source: §1 · p. 3, §7 · p. 15

    The acceptance score predicts accept/reject outcomes but does not measure novelty or long-term impact and provides no explanation for its prediction.

    Source: §8 · p. 15

Abstract

from arXiv · show

Automated research is largely evaluated by benchmark performance, but the value of research rarely reduces to a single score. Humans judge a work's holistic contribution through peer review, which is hard to model because it is noisy and depends on many factors. However, what if, given only the paper, we could predict its peer-review decision directly? We introduce PaperScorer, a family of models that predict a paper's accept or reject decision from its extracted text or page screenshots, training only on the decision after removing author identity and formatting shortcuts. Vision models are the most predictive (68% accuracy on ICLR papers and 77% on ML papers from arXiv), beating text-only and frontier reasoning models. Unlike plain text, page images retain figures, layout, and presentation. This signal is robust, surviving smaller models and even compressing the whole paper into a single image. But accurate predictions are not evenly spread. PaperScorer is strongest on the lowest and highest rated papers, while papers rated in the middle stay unpredictable for our models and frontier systems alike. Additionally, its confidence is calibrated, so its most confident predictions are also its most accurate. Overall, PaperScorer shows how much of the decision a paper carries on its own, marking a first step toward AI-driven research evaluation.

1 Introduction

Peer review is noisy, yet automated research needs a way to judge contributions that benchmarks do not capture; this motivates asking how much a paper alone predicts its final decision. PaperScorer tests that signal using anonymized papers and finds it is strongest in vision, at rating extremes, and when predictions are confident.

1 Introduction

  • Peer review combines reviewer judgments into a final accept/reject decision, but committee experiments found roughly half of papers accepted by one committee would not be accepted by another.The resulting decision mixes paper signal with noise from reviewer assignment, interpretation, aggregation, and AI tools.Source: §1 · p. 2
  • PaperScorer predicts binary acceptance from anonymized papers, comparing markdown text with page screenshots on balanced ICLR and arXiv datasets from 2020–2026.The controls address author identity, formatting shortcuts, and majority-reject bias.Source: §1 · p. 2
  • 68% accuracy on ICLR and 77% on arXiv: PaperScorer-Vision beats the best frontier baseline by 2.7% and 9.0%, respectively.After validation calibration, confidence matches accuracy on test datasets.Source: §1 · p. 2
  • Vision outperforms text on both datasets; accuracy is near chance for intermediate-rated papers, while PaperScorer handles both low- and high-rated papers better than frontier models.Frontier models reliably flag only the lowest-rated papers; vision preserves figures, layout, and presentation that text extraction discards.Source: §1 · p. 2
  • PaperScorer is presented as a first step for estimating how much of a human decision the paper carries, with calibrated estimates intended for agentic reviewing and literature search.The authors connect this need to rising submission volumes and reviewer burden as automated research accelerates.Source: §1 · pp. 2–3

2 Problem Formulation

The paper frames acceptance prediction as estimating the final binary decision from paper artifacts alone, while distinguishing that target from the noisy process that produces it. It also gives a qualified reference point for the accuracy ceiling implied by committee disagreement.

2 Problem Formulation

  • The predictor takes a paper artifact and returns both a hard accept/reject decision and a score whose higher values indicate greater predicted acceptance probability.Acceptance is coded as 1 and rejection as 0.Source: §2 · p. 3
  • The peer-review process is represented as a latent chain from initial submission and reviews through rebuttal, revision, meta-review, and final decision.The notation distinguishes the initial paper X0 from the rebuttal-revised paper Xreb.Source: §2 · p. 3
  • Because anonymous public artifacts are most consistently available after rebuttal, the practical target is p(Z | Xreb), not p(Z | X0), and reviews or ratings are not regression targets.The passage contrasts this setup with prior work (Gao et al., 2024; Zhu et al., 2025).Source: §2 · p. 3
  • The decision-only target is cast as text generation so prompting, supervised fine-tuning, and reinforcement learning can be compared in a common format.When token probabilities are available, the acceptance score is derived from the probabilities assigned to the two decision tokens.Source: §2 · p. 3
  • 80.2% is the estimated Bayes-optimal accuracy on a balanced evaluation under a fitted Beta-Bernoulli model, used only as a soft reference rather than a hard limit.The estimate is based loosely on one historical committee-consistency experiment and inherits its assumptions.Source: §2 · p. 3

3 PaperScorer: Dataset and Methods

PaperScorer is trained and evaluated on balanced, anonymized ICLR and source-derived arXiv paper corpora, with controls intended to make paper content the available signal. The methods compare text and vision models, supervised fine-tuning, reinforcement learning, and prompting baselines.

3.1 Datasets

  • The balanced ICLR and arXiv corpora use 85/5/10 train, validation, and test splits, with evaluation on recent years and ICLR comprising 25K submissions.The larger arXiv training set has 83K papers, alongside a 25K arXiv-m set proportional to ICLR; Figure 2 shows dataset examples.Source: §3.1 · p. 4, Figure 2 · p. 4
  • Corpus inclusion requires anonymity, main-conference scope, and consistent processing so accept/reject artifacts do not differ through formatting or availability shortcuts.ICLR uses latest-rebuttal PDFs for anonymous availability, excludes 2024 because accepts are only camera-ready, and may retain rebuttal-era revisions.Source: §3.1 · p. 4
  • The arXiv corpus derives accepted labels from proceedings matches and defines implied rejects as eligible venue-template papers absent from proceedings, yielding noisier but more diverse data.Its transfer tests compare diverse arXiv supervision with cleaner same-source labels and evaluate cross-source generalization.Source: §3.1 · pp. 4–5
  • Balanced testing weights accept and reject recall equally, while balanced training avoids the excess rejection predictions seen with natural-rate training without per-dataset threshold tuning.Table 1 summarizes accept-minus-reject recall gaps after prior and threshold rules.Source: §3.1 · pp. 5–6, Table 1 · p. 5

3.2 Methods

  • The study uses Qwen2.5 text and Qwen2.5-VL vision backbones, selected to plausibly predate public ICLR 2025 decisions; a Qwen3 text swap changes accuracy little.The authors chose this cutoff logic to make ICLR a cleaner deployment setting.Source: §3.2 · p. 6
  • Comparisons include DeepReview, Gemini, GPT, Jev, and coding-agent harnesses; prompting uses simulated reviews or rating-plus-decision instructions, with held-out score thresholds when they improve accuracy.The main DeepReview setup uses four simulated reviewers, self-verification, and a meta-review decision; frontier ratings are not probabilities.Source: §3.2 · p. 6
  • Supervised fine-tuning is the primary method, varying model scale, training source and prior, text versus screenshot inputs, and additional suffixes; acceptance scores normalize probabilities over Accept and Reject.The sweep includes Qwen2.5 text models at 3B, 7B, and 14B and vision models at 3B and 7B.Source: §3.2 · pp. 6–7
  • The reinforcement-learning variants use GRPO with within-group relative rewards for decision correctness and required output format, allowing reasoning before the final label.The format reward matters early in training, when completions often omit the boxed answer.Source: §3.2 · p. 7

4 Results

PaperScorer’s vision model outperforms frontier baselines on both datasets, while results also show how supervision, scale, and visual information relate to prediction. The section defines evaluation metrics and establishes a simple-feature baseline before analyzing these results.

4.1 Evaluation Metrics

  • Accuracy tests agreement with final decisions, while AUC measures whether accepted papers rank above rejected papers independent of a fixed threshold.The section also distinguishes calibration and class-bias diagnostics from the primary metrics.Source: §4.1 · pp. 7–8, Table 2 · p. 7

4.2 A Simple Paper Features Baseline

  • 61.1% accuracy on ICLR and 62.5% on arXiv came from logistic-regression probes using hand-crafted paper features.Accept-associated features generally involved paper length, references, and figures; reject-associated features were more lexical.Source: §4.2 · p. 8

4.3 Main Results

  • 68.3 Acc on ICLR and 77.5 on arXiv, PaperScorer-Vision exceeded the best frontier baseline by 2.7% and 9.0%, respectively.Table 3 compares prompting baselines, trained reviewers, and PaperScorer variants on both datasets.Source: Table 3 · p. 9, §4.3 · p. 10
  • Across years, matched-source supervised PaperScorer led Acc/AUC, vision was strongest, prompting followed, and RL trailed prompting.Direct supervision on final decisions worked better than learning from long reasoning traces with a single noisy decision reward.Source: Figure 5 · p. 10, §4.3 · p. 10
  • Training only through ’25 transferred poorly to ’26, where accuracy and AUC dropped across PaperScorer, frontier models, and DeepReview.The authors attribute the shared drop to a change in evaluation distribution and expect periodic retraining as new conferences arrive.Source: §4.3 · p. 10
  • Decision accuracy changed little across 3B, 7B, and 14B models, while more training data produced the largest arXiv gain.The arXiv pool increase was 3.4×; the smaller ICLR dataset makes its data-size curves unsuitable for absolute comparison with arXiv.Source: §4.3 · p. 11, Figure 6 · p. 11
  • Vision improved PaperScorer over text on both datasets, with paired-bootstrap gains significant after correction.Screenshots retain figures, layout, and visual presentation that markdown extraction discards; frontier-model vision–text gaps were small and not significant after thresholding.Source: §4.3 · p. 11
  • Compressing ten vision pages into one image cut tokens by about 63% and reduced ICLR performance by only 2.2%.OCR retention fell from about 80% on page screenshots to about 30% on the compressed panel, while layout and presentation remained.Source: §4.3 · p. 11, Table 4 · p. 12

5 Where does this signal lie?

PaperScorer’s confidence tracks its accuracy after temperature scaling, but predictability varies across papers and venues. Models are weakest on middle-rated papers, while PaperScorer-Vision remains accurate at both rating extremes.

Confidence tracks accuracy

  • After scaling, PaperScorer confidence matches accuracy on both datasets; at confidence of at least 70%, it covers about a third of ICLR papers and over half of arXiv papers, at nearly 80% and 85% accuracy.Jev covers only 6% and 12% at that confidence threshold.Source: Figure 8 · p. 12, §5 · p. 13

Deployment at the real acceptance rate

  • At ICLR’s natural acceptance rate, Accept calls were correct 47% of the time and Reject calls 82%.Adding the venue’s estimated acceptance prior reduced calibration error from .13 to .02–.03.Source: §5 · p. 13

Selective reliability

  • Across models, accuracy is near chance on middle-rated papers; PaperScorer-Vision stays accurate at the highest ratings, unlike GPT and Gemini.Accuracy peaks on confident rating extremes and collapses in the middle, where none of the evaluated models finds enough signal.Source: §5 · p. 13, Figure 9 · p. 13

Performance across conferences

  • On six of seven high-coverage venues, PaperScorer-Vision matched or beat the best frontier model; Gemini tied it on CVPR and won on ICCV.On AAAI, PaperScorer was near 90 while both frontier models fell close to chance; the authors report this was not a length effect.Source: §5 · pp. 13–14, Figure 10 · p. 14

6 Related Work

Related work spans paper-outcome prediction, reviewer-agent systems, and model-based evaluation in automated scientific workflows. PaperScorer is positioned as a calibrated decision predictor amid concerns about label provenance and decision priors.

Paper acceptance prediction and dataset construction

  • Earlier paper-outcome prediction used bibliometric, lexical, and handcrafted signals [Zhao et al., 2025; Hull and Bihlo, 2025], while newer systems move toward paper-level accept/reject prediction.Prior datasets can leak labels through source or revision artifacts; this work constructs datasets where labels are recoverable only from paper content.Source: §6 · p. 14

Review generation and reviewer agents

  • Review-generation and agentic-review systems target critiques or review simulation rather than direct final-decision prediction; examples include DeepReview [Zhu et al., 2025] and AgentReview [Jin et al., 2024].The passage motivates calibrated acceptance probabilities as decision priors for agentic reviewing.Source: §6 · p. 14

AI-assisted scientific workflows

  • Automated scientific workflows use model-based evaluation to filter ideas or steer search, including The AI Scientist [Lu et al., 2024; Yamada et al., 2025] and FunSearch.The passage frames reviewer models as selection mechanisms whose bias or calibration can shape pursued research directions.Source: §6 · p. 15

7 Conclusion and Applications

PaperScorer predicts final accept/reject decisions from paper content, with vision models outperforming frontier reasoning and DeepReview baselines. Its calibrated acceptance score also improves literature retrieval and agentic reviewing, but it is neither a replacement for peer review nor a universal measure of scientific value.

7 Conclusion and Applications

  • 7B vision PaperScorer models outperform frontier reasoning and DeepReview baselines on decision accuracy and ranking while producing calibrated acceptance probabilities.Screenshots preserve layout, figure, and presentation cues that markdown extraction tends to discard.Source: §7 · p. 15
  • 8.7% higher ground-truth citation recall in LitSearch top-10 results follows when semantic retrieval scores are combined with PaperScorer-Vision’s acceptance score.Conditioning Claude Code on the PaperScorer-Vision prior raises decision accuracy from 63.0% to 70.6%, mainly by correcting incorrect acceptance predictions.Source: §7 · p. 15

8 Limitations and Future Work

The authors identify limits in explaining predictions, generalizing to newer papers, and modeling borderline cases. The appendix documents supporting materials and derives a Bayes-optimal accuracy reference from committee-consistency data.

8 Limitations and Future Work

  • The model’s attention maps and section-edit tests do not establish why it favors one paper, and the proposed role of overall visual appearance remains conjectural.The analysis uses last-layer attention from only a few papers.Source: §8 · p. 15
  • Performance weakens on newer papers because the data are thin; ’26 is also unusual in rebuttal length, rating gaps, and arXiv venue coverage.The authors expect this limitation to ease as more ’26 papers appear and note that the model can be retrained through their pipeline.Source: §8 · p. 15
  • The acceptance score predicts decisions, not novelty or long-term impact, and provides no explanation; its citation alignment evidence is preliminary, based on ICLR ’25 alone.On that slice, acceptance probability tracks later citations about as well as the human review score (ρ ≈.25 for both).Source: §8 · p. 15
  • Future work targets broader high-fidelity datasets, better borderline-paper modeling, and richer supervision than one binary outcome, especially for disagreement and venue-specific norms.The appendix supplies derivations, dataset documentation, extended evaluation, and training and inference details.Source: §8 · p. 15, Appendix · p. 20

A Bayesian Optimal Accuracy Derivation

  • Under the natural acceptance distribution, the Bayes-optimal predictor accepts when p > 0.5, yielding an accuracy ceiling of approximately 83.3%.The derivation estimates a Beta distribution from the reported acceptance and committee-disagreement rates.Source: A Bayesian Optimal Accuracy Derivation · p. 22
  • For the balanced evaluation pool, the optimal decision threshold shifts to p > 0.26 under a uniform 50/50 class prior.The balanced ceiling is obtained by averaging over the conditional Beta distributions.Source: A Bayesian Optimal Accuracy Derivation · p. 22

B.1 Shortcuts in Prior Datasets

The appendix audits leakage shortcuts in existing datasets and documents how PaperScorer’s ICLR and arXiv datasets are selected and constructed. It excludes ICLR ’24 because masking identity does not remove residual camera-ready layout cues.

B.1 Shortcuts in Prior Datasets

  • Prior accept/reject datasets can leak labels through source, venue, or revision artifacts, enabling prediction from formatting rather than paper content.Deep Paper Gestalt pairs accepted CVPR papers with workshop rejects whose shorter length and distinct headers can provide a shortcut.Source: §B.1 · p. 22, Table B.1 · p. 23
  • DeepReview ’24 carries source for 78.7% of accepts but 41.2% of rejects, while AI Scientist ’22 exposes author names on 100% of accepts and essentially no rejects.The selected PaperScorer years stay at most 0.92% A / 1.73% R on any tested signal; the audit found no such shortcuts in latest pre-camera-ready rebuttal artifacts.Source: §B.1 · p. 23, Table B.2 · p. 23

B.2 Data Provenance and Ensuring Anonymity

  • ICLR ’24 is excluded because camera-ready accepts paired with rebuttal rejects retain layout differences after identity spans are removed and pages whitened.The introduction start position differs notably between accepts and rejects in ’24, consistent with residual layout cues detected by vision models.Source: §B.2 · p. 24, Figure B.1 · p. 24
  • Reviewer rating scales change across ICLR years, so ratings are normalized to percentile space; explicit year-specific length filters account for changing page limits.The scale shifts from 4 points in ’20 to 10 in ’21 and 6 from ’22 onward.Source: §B.2 · p. 24, Table B.4 · p. 24

B.3 Our Data Pipeline

  • The ICLR and arXiv workflows independently construct datasets: ICLR uses OpenReview decisions, while arXiv recovers venue and decision from source tarballs and public paperlists.The ICLR workflow is simpler because OpenReview publishes decisions for both accepts and rejects; the arXiv workflow must recover both labels and venues.Source: §B.3 · p. 25, Figure B.2 · p. 25
  • The ICLR pool contains 24,888 papers, while the full arXiv pool contains 83,656 papers across multiple ML venues; arXiv-m is a 24,650-paper compute-matched subsample.Both workflows apply anonymization and balanced stratified splits; ICLR excludes ’24.Source: §B.3 · pp. 25–26, Table B.5 · p. 25

C Simple Feature Analysis

The feature analysis tests whether simple structural and textual signals separate accept from reject papers. Its strongest individual feature is far weaker than PaperScorer end to end, and most probe coefficients are small.

C Simple Feature Analysis

  • The inventory defines the probe features computed from concatenated human-side paper text truncated to the first 80,000 characters.Feature names marked “per pg.” are normalized by an estimated body-page count.Source: Table C.1 · p. 26
  • Paper length is the strongest individual signal on both sources at w ≈+0.48, while reject-side magnitudes are |w| ≤0.25 and most features fall between −0.05 and +0.10.These probe signals are an order of magnitude below PaperScorer’s end-to-end performance, indicating that simple structural shortcuts do not cleanly separate decisions.Source: C Simple Feature Analysis · p. 30
  • The logistic-regression probes use signed standardized coefficients against ICLR and arXiv decision labels, excluding source identifiers, venue labels, ratings, citations, author names, and non-comparable page counts.The main text reports the top five accept- and reject-associated features for each source.Source: C Simple Feature Analysis · p. 30

D Do Frontier Models Memorize the Test Decisions?

The authors test whether frontier-model accuracy reflects memorized test decisions, compare text backbones, and examine balanced evaluation against natural-rate performance. GPT-5.5 recognizes many papers but its recognition does not explain its accuracy; Qwen backbone accuracy is close, and balanced-test metrics closely track natural-rate accuracy after reweighting.

D Do Frontier Models Memorize the Test Decisions?

  • 50–73% is how often GPT-5.5 is correct when it claims to recall a decision, but its claimed decisions have 44% balanced accuracy on ICLR.Its ICLR accept bias and near-zero abstract-continuation ROUGE-L further weaken the case that it retrieves test decisions.Source: D Do Frontier Models Memorize the Test… · p. 30
  • Recognized papers are no easier for GPT-5.5 to classify, including ICLR ’25, and accuracy does not rise with the number of recognition cues.The authors conclude that memorization does not explain GPT-5.5’s baseline accuracy.Source: Table D.2 · p. 31, D Do Frontier Models Memorize the Test… · p. 31

E Backbone Choice: Qwen2.5 vs. Qwen3

F.1 Evaluation and Training Prior Investigation

  • Balanced-test accept and reject recalls reweighted by the natural accept prior predict natural-rate accuracy within a few percentage points across four matched cells.The authors report that this keeps balanced-test accuracy aligned with their decision objective.Source: Table F.1 · p. 32, §F.1 · p. 32

F.2 Ablating Training Distribution and Paper Representation

Training-distribution and representation ablations favor the simple paper-only training pool for vision, while added context can help text but not the strongest vision model. Reverse-order performance changes little; cross-source transfer is more difficult, with models tending to reject unfamiliar papers and losing ranking quality under the shift.

F.2 Ablating Training Distribution and Paper Representation

  • Extra context can help text modestly, especially on ’26, but does not improve the strongest vision model; the selected paper-only pool gives the strongest vision results without sacrificing text performance.Rating-agreeing data helps on ’25 but does not noticeably improve either modality once ’26 is included; ’26 has the smallest accept-reject rating gap.Source: Table F.2 · p. 33, §F.2 · p. 33
  • Paper statistics and dummy reasoning tokens do not improve accuracy; synthetic reviews sometimes help on ICLR ’25 but hurt on ICLR ’26.Combining full text and screenshots also fails to outperform single-modality baselines, and the added context may increase noise without adding complementary signal.Source: §F.2 · p. 33

F.3 Reverse-Order Ablation

  • 1–1.3 points of accuracy are lost when section blocks or page order are reversed, while forward-trained checkpoints remain close to forward accuracy on reversed inputs.The authors characterize vision in particular as largely order-invariant.Source: §F.3 · p. 33, Table F.3 · p. 34

G Cross-Source Generalization

  • On the other source’s full test set, each 7B vision model becomes reject-predictive and recovers only about half of the true accepts.Retuning the threshold swaps errors without improving accuracy; the authors report that the paper ranking, not the threshold, fails under shift.Source: Table G.1 · p. 34, G Cross-Source Generalization · p. 34

H Prompting and Extended Experiments

The appendix details prompt configurations, prompting and RL experiments, and calibration procedures. Prompt variants do not lift accuracy beyond chance, while base-rate-adjusted calibration reduces probability error; the RL reward curves also mask persistent accept bias.

H.1.1 Base Open-Source Prompting; H.1.2 Frontier Model Prompting

  • Open-source prompts request a boxed accept/reject decision, while frontier prompts request a rating plus decision; frontier models use a venue-generic prompt.After SFT, decoding is greedy; the reported prompt configurations and context limits are summarized in Table H.1.Source: §H.1.1 · p. 35, Table H.1 · p. 35, §H.1.2 · p. 36

H.2 Frontier Model Reasoning Usage

  • 1.3k–2.3k median reasoning tokens per paper on text inputs and 1.5k–2.1k on vision inputs are reported for frontier models.P95 reasoning length reaches 3.6k–4.7k tokens; ICLR cells use more reasoning than matched arXiv cells.Source: Table H.2 · p. 37, §H.2 · p. 37

H.3 Correcting Model Biases with Prompting

  • ∼50% ICLR accuracy remains the ceiling across prompting variants, even when multi-step or structured prompts move accept and reject recall.The Critical modifier with Structured JSON reaches 31.8% reject recall for text and 32.1% for vision, but accuracy stays at chance.Source: §H.3 · pp. 37–38, Table H.3 · p. 38

I.1 SFT; I.4 Hardware Details

  • SFT uses cosine decay, five warmup steps, and a 24,480-token context; the strongest text and vision configurations use batch sizes 32 and 16, respectively.The reported SFT and RL experiments run on one node with 4× NVIDIA H200 GPUs.Source: §I.1 · p. 39, §I.4 · p. 41

I.2 RL

  • Graded reward rises during RL, but the accept bias persists and tightens, and neither policy surpasses the strongest SFT model.The 0.2 format-reward floor can make reward gains obscure the limited change in the accept/reject mix.Source: §I.2 · p. 39, Figure I.1 · p. 39

I.3 Calibration Details

  • Temperature scaling with a base-rate term adjusts probabilities for the venue’s acceptance rate, while τ controls their sharpness and does not change decisions.A two-parameter Platt fit is slightly better still; Jev’s calibrated probabilities never exceed about 60%.Source: §I.3 · p. 40
  • Natural-rate calibration uses ten equal-width bins of acceptance probability to compare mean predicted acceptance with the observed accepted fraction in each bin.The appendix also notes that arXiv’s approximately 23% prior mixes venue rates and that its rejects are inferred rather than confirmed against proceedings.Source: §I.3 · p. 40
  • .023–.042 ECE for PaperScorer follows base-rate shifting, compared with .13–.15 without the shift on natural-rate tests.With the shift, PaperScorer-Vision’s accept calls are correct 47% of the time on ICLR and 56% on arXiv; reject calls are correct 82% and 91%, respectively.Source: §I.3 · p. 40, Table I.1 · p. 41

J Statistical Uncertainty

The appendix tests statistical uncertainty, applies PaperScorer’s acceptance estimates to literature retrieval and agentic reviews, and examines which paper sections influence its predictions.

J Statistical Uncertainty

  • On ICLR, Acc gaps were +3.8% against Gemini-3.1-Pro and AUC gaps +.066 against GPT-5.5; on arXiv, the respective gaps were +8.7% and +.097, all significant.The reported gaps are paired-bootstrap comparisons; all four directional tests had p ≤ .014.Source: J Statistical Uncertainty · p. 41, Table J.1 · p. 42
  • Vision beat text in five of six comparisons; PaperScorer’s vision model was significantly better on both datasets, while frontier-model gaps were not significant.For PaperScorer, the corrected one-sided tests gave p = .012 and .005.Source: J Statistical Uncertainty · p. 42, Table J.2 · p. 42
  • On ICLR, PaperScorer-Vision’s reviewer-rating alignment was ρR = +.48; Gemini-3.1-Pro’s repeated decisions agreed 89.4%, with its second pass 4.5% lower.The repeatability audit used GPT-5.4, not the later GPT-5.5 baseline.Source: J Statistical Uncertainty · p. 42, Table J.3 · p. 43

K.1 Ranking Scientific Literature for Retrieval

  • On 148 hard LitSearch queries, blending retrieval scores with PaperScorer-Vision’s acceptance estimate improved both BM25 and Qwen reranking, with the largest gains near the top.The acceptance estimate adds a prior to retrieval rather than replacing topical ranking.Source: Figure K.1 · p. 43, §K.1 · p. 44

K.2 Grounding Agentic Reviewers

  • Conditioning Claude Code on PaperScorer-Vision raised decision accuracy from 63.0% to 70.6% and improved human-review alignment except for contribution alignment.The prior chiefly corrected Accept→Reject flips, which had 77.9% accuracy on the 13.6% subset of papers that flipped.Source: Table K.1 · p. 44, §K.2 · p. 45

K.3 Agentic Reviewer Prompts

  • Codex grades generated reviews against human reviews on contribution identification, top-weakness identification, novel concerns, and overall review quality.The rubric asks the grader to compare quoted human and agent review text before assigning scores.Source: §K.3 · p. 46

L.1 Section-Attention Map

  • For borderline accepted papers, later-section edits showed the largest observed acceptance-probability changes; effective edits varied by section, from formalization in Methodology to narrative in Experiments and Discussion.Nuance was the most effective edit type in the Introduction.Source: Figure L.1 · p. 47, §L.1 · p. 47

L.2 Interpretability Details

  • Attention maps are descriptive; normalized final-decision-token attention is aggregated by section, while single-section edits test whether changes shift predicted acceptance.The authors caution that attention can reflect decisive evidence, earlier computations, or interactions across sections.Source: §L.2 · p. 48, Figure L.2 · p. 48

L.3 Edit Analysis

Section-level edits broadly mirrored the attention priorities, while the edit types that increased acceptance probability depended on the section.

L.3 Edit Analysis

  • On 30 accepted ICLR ’26 papers with low presentation scores, Experiments/Results and Discussion/Conclusion edits gained 5.4–6.1% paccept, the largest gains across sections.The edit-driven acceptance changes followed the same section ordering as the attention pattern.Source: §L.3 · p. 49
  • Introduction edits favored Nuance (+11%) and Calibration (+5%) but Narrative lowered scores (−13%); Background favored Substantiation and Narrative (+8% each), while Nuance was penalized (−12%).Source: §L.3 · p. 49
  • Methodology favored Formalization (+14%); Narrative helped Experiments (+12%) and Discussion (+24%), while Discussion Substantiation was most detrimental (−29%).The authors interpret these edit-success distributions as reflecting general paper guidelines.Source: §L.3 · p. 50

Related papers

Loading 2610.31791v1…