PaperlayerView exact revision on arXiv

Source-linked AI summary

ScholarPeer: A Context-Aware Multi-Agent Framework for Automated Peer Review

Palash Goyal, Mihir Parmar, Yiwen Song, Hamid Palangi, Tomas Pfister, Jinsung Yoon

arXiv:2601.22638v2cs.MAcs.AIcs.LG

TL;DR

Machine-learning submission growth strains peer review, with slow author feedback and heavy reviewer auditing demands. ScholarPeer combines specialized agents to contextualize and interrogate papers, and on ∼1,800 ICLR submissions it outperforms the reported baselines with an H-Max score of 6.12.

  • Problem

    Rapid growth in machine-learning submissions leaves authors waiting months for feedback while reviewers face burdensome audits of technical soundness, baselines, and literature claims.

    Source: §1 · p. 1

  • Method

    ScholarPeer uses specialized agents to structure paper content, build external context, find omitted baselines, and interrogate claims to support human review.

    Source: §3.1 · p. 4, §6 · p. 11

  • Results

    H-Max 6.12, with ScholarPeer decisively outperforming state-of-the-art fine-tuned models and search-augmented agentic baselines on ∼1,800 ICLR submissions.

    Source: §6 · p. 11

  • Takeaways & Limitations

    ScholarPeer is intended to accelerate author iteration and augment human reviewers with actionable, verifiable evidence rather than replace human judgment.

    Source: §6 · p. 11

    The Stanford Agent Reviewer comparison used browser-based access and evaluation on only 50 sampled ScholarEval papers.

    Source: §4.1 · p. 6

Abstract

from arXiv · show

The exponential growth of machine learning submissions has strained the traditional peer review process, resulting in slow feedback loops for authors and an immense burden on reviewers to rigorously audit technical soundness and verify literature. To address this, we introduce ScholarPeer, a multi-agent framework designed to operationalize the rigorous auditing workflow of a senior researcher. Rather than attempting to replace human judgment, ScholarPeer serves as a co-scientist: acting as a mentor for rapid author iteration prior to submission, and as an active verification assistant that augments human reviewers. The framework structurally decouples contextualization from critique by deploying a sub-domain historian to synthesize the field's trajectory, a baseline scout to proactively hunt for omitted state-of-the-art comparisons, and a multi-aspect Q&A engine that deeply audits technical soundness-scrutinizing internal logical consistency, experimental validity, and mathematical rigor-while cross-referencing claims against top-tier academic venues. We comprehensively evaluate ScholarPeer on ~1,800 ICLR submissions spanning 2020 through 2025. Our results show that ScholarPeer achieves significant win-rates against state-of-the-art fine-tuned models and search-augmented agentic baselines.

1. Introduction

ScholarPeer is introduced as a co-scientist to support author iteration and reviewer verification amid a strained review process. The paper reports broad baseline comparisons on a large ICLR submission set.

1. Introduction

  • Machine-learning submission growth strains peer review: authors wait months for feedback while reviewers must audit soundness, find missing baselines, and verify state-of-the-art claims.The paper describes major conferences as receiving tens of thousands of submissions annually.Source: §1 · p. 1
  • ScholarPeer structurally separates paper comprehension from external contextualization, using a historian, baseline scout, and skeptical Q&A engine to audit claims and evidence.Its stated role is to augment, not replace, human review, supplying verifiable evidence to experts.Source: §1 · pp. 1–2
  • The evaluation covers ∼1,800 ICLR submissions from 2020 through 2025 and introduces H-Max for critique quality and review diversity for variance of perspectives.The dataset merges the evaluation sets of DeepReview-Bench and AgentReview.Source: §1 · p. 2
  • ScholarPeer achieves dominant win-rates against state-of-the-art fine-tuned models and search-enabled agentic baselines, with reviews costing ∼$1.20 and taking ∼10 minutes.The framework also consistently outperforms baselines across various metrics.Source: §1 · p. 2, Figure 1 · p. 2

2. Problem Formulation

The paper formulates automated peer review assistance as producing a rigorous, verifiable critique by combining a submission's intrinsic content with the field's changing context.

2. Problem Formulation

  • The task is to generate a rigorous, verifiable critique R for a paper submission S.The formulation separates paper-internal claims and experimental setup from the scientific field's evolving context.Source: §2 · p. 3
  • The intrinsic submission content comprises its claims and experimental setup, while the external context includes prior art, concurrent work, and community standards.The formulation likens this external context to a human reviewer's dynamic mental graph.Source: §2 · p. 3
  • ScholarPeer implements the mapping through multi-agent orchestration that constructs dynamic context, transforms submission content, and actively interrogates the paper.The resulting review is designed to augment human judgment and accelerate author iteration.Source: §2 · p. 3

3. ScholarPeer

ScholarPeer uses a dual-stream architecture to structure paper comprehension and external context before generating a review. Specialized agents acquire context, probe the paper, and synthesize findings under review guidelines.

3. ScholarPeer

  • The architecture divides processing between knowledge acquisition and contextualization and a multi-aspect Q&A engine, rather than relying on a single manuscript pass.The design is presented as addressing lost-in-the-middle effects and surface-level summaries.Source: §3 · p. 3

3.1. Knowledge Acquisition and Contextualization

  • The summary agent turns the submission into a structured representation of core claims, method, and reported evidence for downstream agents.This separates structural comprehension from critique and is intended to give downstream agents a high-fidelity signal of the paper's premises.Source: §3.1 · p. 4
  • The literature agent builds dynamic context through an initial sub-domain search followed by expansion searches for gaps, recent preprints, and concurrent work.It filters sources to selected top-tier conferences and arXiv, applies the paper's submission cutoff, and retrieves verifiable literature through a search API.Source: §3.1 · p. 4
  • The sub-domain historian compresses retrieved literature into a chronological domain narrative to situate contributions along the field's trajectory.Source: §3.1 · p. 4
  • The baseline scout independently searches benchmark-specific state-of-the-art methods and returns potentially omitted competitors and datasets.Its findings provide evidence for technical-soundness critique.Source: §3.1 · p. 4

3.2. Multi-Aspect Q&A Engine and Review Generation

  • Dedicated Q&A agents probe novelty and significance and technical soundness using the structured summary, domain narrative, and baseline findings.Novelty answers use active search; soundness answers check claims against the paper's methodology and evidence for inconsistencies, mathematical constraints, and evaluation metrics.Source: §3.2 · p. 4
  • The review generator combines the structured summary and verified Q&A pairs under explicit venue guidelines to produce a constructive, venue-aligned critique.The design separates investigation in the Q&A engine from reporting in the review generator.Source: §3.2 · pp. 4–5

4. Experiments

Experiments evaluate ScholarPeer on a longitudinal ICLR benchmark using side-by-side judgments and complementary measures of critique quality, diversity, and alignment with human rankings. ScholarPeer wins against fine-tuned and agentic baselines, scores highly against human reviews, and shows gains across its agentic components.

4.1. Evaluation Protocols

  • Evaluation triangulates critique quality through five-dimension side-by-side judgments, H-Max against the strongest human points, review diversity, and decision-score correlation with human rankings.H-Max compares a review with the strongest human points on an aspect; review diversity is 1 minus inter-review semantic similarity.Source: §4.1 · p. 5

4.2. Experimental Setup

  • ScholarEval merges ICLR submissions from 2020–2025 (1,786 total); ScholarPeer reviews averaged 597 words, compared with 788 for human reviews and 2,156 for DeepReviewer-14B.Source: §4.2 · pp. 6–7

4.3. Side-by-Side Results

  • 90.5% win-rate against DeepReviewer-14B, 73% against the strongest same-backbone agentic baseline, and 54% against Stanford Agent Reviewer.With the same Google Search tool, ScholarPeer also outperforms search-augmented AI Scientist v2 and AgentReview; a Claude-backbone reverse ablation reports 90.2%, 79.6%, and 78.8% win rates against DeepReviewer-14B, AI Scientist v2, and AgentReview, respectively. Blinded experts preferred ScholarPeer across five baselines.Source: §4.3 · pp. 7–8

4.4. H-Max Results

  • 6.12 average H-Max accompanies scores of 6.47 in Significance Assessment and 5.87 in Constructive Value, with standard deviation 0.01 across three runs.More than 25% of generated reviews scored 8.0 or above; ScholarPeer also achieved the highest average rating across human evaluation and the LLM judge.Source: §4.4 · p. 8, Figure 3 · p. 8

4.6. Summary of Gains

  • The qualitative comparison finds ScholarPeer strongest in external SOTA verification and contextual depth, while AI Scientist v2 remains competitive on internal consistency checks.The comparison summarizes LLM-judge reasoning traces.Source: Figure 4 · p. 9, §4.6 · p. 9

4.5. Review Diversity

  • RDS = 0.31 gives ScholarPeer the highest review diversity among automated systems, versus 0.43 for human committees and 0.01–0.02 for fine-tuned models.Single- and multi-agent baselines reach up to 0.25; the authors hypothesize that varied literature-expansion paths and Q&A probes contribute to ScholarPeer’s diversity.Source: §4.5 · p. 9

4.7. Ablation, Hyperparameter Sensitivity & Deployment Practicality

  • Removing Q&A causes the largest reported win-rate drop (26%), versus 9% for removing the baseline scout and 10% for removing the summarizer.Gains saturate after 10 Q&A pairs and 3 search rounds; deployment is approximately $1.20 and 10 minutes per paper, or about 56 seconds per review at batch size 20.Source: Figure 5 · p. 9, §4.7 · p. 10

5. Related Work

Automated review research has moved from fine-tuned models toward agentic and retrieval-augmented systems, while evaluation work increasingly addresses actionability and risks such as homogenized opinions.

LLM-based Automated Reviewing

  • DeepReviewer uses chain-of-thought fine-tuning to improve reasoning, while other systems fine-tune for review style or venue norms.CycleReviewer uses negative constraints; OpenReviewer and ReviewAgents target venue-specific norms.Source: §5 · p. 10

Agentic & Retrieval-Augmented Frameworks

  • Agentic and retrieval-augmented approaches include multi-stage review discussion, preference-aligned reviews, graph-based information retrieval, and agentic scientific Q&A.Agent Review models review as multi-stage discussion; ReviewRL and REMOR use reinforcement learning, AutoRev uses hierarchical graphs, and PaperQA2 uses agentic RAG.Source: §5 · p. 10

Foundations of Automated Survey & Verification

  • Survey-generation frameworks inform the historian, while chain-of-verification and scientific fact-checking benchmarks inform active searches for omitted evidence and baselines.The cited examples include SurveyG, AutoSurvey, SciFact, and Co-Sight.Source: §5 · p. 10

Evaluation of AI Reviewers

  • Evaluation efforts target critique actionability and multimodal review, while this work measures human-calibrated actionability and review diversity to assess homogenization.Prior work also identifies LLM-judge bias and the risk of an “artificial hivemind.”Source: §5 · pp. 10–11

6. Conclusion

ScholarPeer separates paper comprehension from external contextualization and uses specialized agents to support actionable review and author iteration. Its evaluation reports strong critique results, while the impact discussion highlights deployment risks and keeps final decisions with people.

6. Conclusion

  • ScholarPeer decouples paper comprehension from contextualization, with agents synthesizing domain trajectories, finding omitted baselines, and interrogating technical soundness.It is intended to accelerate author iteration and provide reviewers with actionable, verifiable evidence rather than replace human judgment.Source: §6 · p. 11
  • On ∼1,800 ICLR submissions from 2020–2025, ScholarPeer reports an H-Max score of 6.12 and outperforms fine-tuned models and search-augmented agentic baselines.The conclusion says future work will explore collaborative multi-agent debate to deepen critique rigor, diversity, and actionability.Source: §6 · p. 11

A. Impact Statement

  • The impact statement presents ScholarPeer as a way to address publishing-scale pressures and broaden access to actionable feedback, particularly for authors lacking senior mentorship.These are stated aims of the work.Source: A. Impact Statement · p. 14
  • Automated critiques risk over-reliance, while automated systems may penalize novel ideas and narrow acceptable scientific inquiry; search-enabled deployment also raises confidentiality concerns.The authors call for secure, ephemeral handling of submissions and no use of submission data to train public models.Source: A. Impact Statement · p. 14
  • ScholarPeer is intended as an evidence-grounded second opinion, with final scientific decisions remaining in human hands.The authors explicitly frame the system as augmenting rather than replacing human judgment.Source: A. Impact Statement · p. 14

B. Limitations

The limitations section identifies a gap between system and human review diversity and higher latency and cost from active web-scale retrieval; it also notes that configuration details are provided for reproducibility.

B. Limitations

  • ScholarPeer’s review diversity score of 0.31 remains below the 0.43 variance of human expert reviews.The authors identify closing this gap as important for capturing the full spectrum of scientific opinion.Source: B. Limitations · p. 14
  • Active web-scale retrieval brings higher inference latency and cost than static fine-tuned models.Source: B. Limitations · p. 14

C. Experiment Details

C.1. Models and Compute

Experiments used Vertex AI, Gemini 3.1 Pro, and specified compute and search configurations.

C.1. Models and Compute

  • All ScholarPeer experiments ran on Google Cloud Vertex AI.Source: §C.1 · p. 14
  • Gemini 3.1 Pro served as the primary ScholarPeer reasoning engine, with generation temperature set to 0.7.Source: §C.1 · p. 15
  • Agentic frameworks used standard CPU instances with Vertex AI model endpoints; fine-tuned models used 8X NVIDIA A100 80GB GPUs.Gemini models used Vertex AI’s native google_search tool, while Claude models used web_search_20250305.Source: §C.1 · p. 15

C.2. Inter-Judge Agreement and Correlation

The section compares LLM judges on overall side-by-side agreement and H-Max score correlation. Gemini 3.1 Flash and Pro agree and correlate most, while Claude Sonnet 4.5 has lower agreement and correlation with Gemini models.

C.2. Inter-Judge Agreement and Correlation

  • 77% is the agreement between Claude Sonnet 4.5 and Gemini 3.1 Pro, the lowest judge-pair agreement; Gemini 3.1 Flash and Pro have the highest.The section reports that the agreement remains high despite being the lowest pairwise rate.Source: Table 4 · p. 15, §C.2 · p. 15
  • 0.66 is the H-Max score correlation between Claude Sonnet 4.5 and Gemini 3.1 Pro; Gemini 3.1 Flash and Pro have the highest correlation.Source: Table 5 · p. 15, §C.2 · p. 15

D. Computational Complexity and Deployment Analysis

ScholarPeer's deployment analysis reports a per-review cost of about $1.20 and an amortized latency of about 56 seconds with parallel batching. The section also details inference-call overhead, token use, and examples highlighting missing-baseline detection and novelty verification.

D.1. Theoretical Inference Cost

  • ∼20 LLM calls per review result from the configuration of 3 literature-expansion rounds and 10 probing questions, versus 1 call for static fine-tuned models.The paper also contrasts this with 5–10 calls for standard agentic baselines.Source: §D.1 · p. 16

D.2. Empirical Token Usage and Monetary Cost

  • ∼429K input tokens and ∼26K output tokens are consumed per average ScholarPeer review across the ScholarEval test set.Source: §D.2 · p. 16

D.3. Latency and Scalability

  • ∼$1.20 per review and ∼56 seconds amortized latency are reported for ScholarPeer, which the authors describe as practical for deployment.The 56-second figure is based on processing papers in parallel batches using 20 concurrent threads.Source: §D.2 · p. 16, §D.3 · p. 16

E. Qualitative Examples

E.1. Example 1: Detection of Missing Baselines

  • The cited paper is titled DBRNet: Advancing Individual-Level Continuous Treatment Estimation through Disentangled and Balanced Representation; the passage does not identify Example 1 or report its findings.Source: §E.1 · p. 17

Baseline Review (DeepReviewer-14B)

The baseline-review examples present ScholarPeer as producing more specific technical and literature critiques than the compared reviews. They cover both DBRNet and a GAN-based federated-learning method, alongside strengths and limitations identified in those papers.

Baseline Review (DeepReviewer-14B)

E.2. Example 2: Technical Soundness

  • GAFM pairs a GAN label generator and discriminator with randomized cross-entropy; the review reports a better privacy-utility trade-off than Marvell and Max Norm on binary tasks.The review also notes two proposed gradient-based attacks, called Mean and Median attacks.Source: Baseline Review (AI Scientist v2) · p. 21

Baseline Review (AI Scientist v2)

  • The GAFM review notes that its defense lacks theoretical guarantees and sufficient computational analysis; its attacks assume known class gradient centers or medians.It also says tuning parameters on a data subset may be difficult in strict VFL settings without violating privacy constraints.Source: Baseline Review (AI Scientist v2) · pp. 21–22

Limitations:

The review identifies missing formal privacy guarantees and a binary-only scope as core limitations, alongside concerns about baseline coverage, stability, and computational overhead.

Limitations:

  • The method has higher computational complexity than simple noise-addition methods.Source: Limitations · p. 22
  • Reviewers ask how runtime and memory overhead, secure hyperparameter tuning, and clustering attacks without ground-truth statistics affect the method.Source: Limitations · p. 22
  • GAFM provides no formal privacy guarantee and relies on heuristic mutual gradient perturbation supported by empirical validation.The review contrasts this with Marvell’s theoretical leakage bound.Source: Limitations · p. 22
  • The mechanism is limited to binary classification, whose opposing class-gradient geometry does not directly extend to multi-class settings.The paper explicitly acknowledges this scope limitation.Source: Limitations · p. 22
  • The evaluation covers perturbation-based defenses but omits relevant non-perturbation comparisons such as CoAE and gradient quantization.The review also proposes these methods as missing baselines.Source: Limitations · pp. 22–23
  • GAN instability and sensitivity to the GAN/CE balance may undermine stability claims; the review says leakage recurs when the balance is off.The authors propose gradient normalization as a heuristic fix.Source: Limitations · p. 23

E.3. Summary of Gains and Losses

This section summarizes comparative assessments of ScholarPeer’s reviews and analyses of GAFM, including reported strengths, weaknesses, and limits of the discussed benchmarks. It covers literature verification, methodological critique, internal consistency and reproducibility checks, and the unproven applicability beyond binary classification.

E.3. Summary of Gains and Losses

  • ScholarPeer’s recurring strengths were relevant literature and SOTA checks, forensic detection of experimental flaws, faithful paper reading, actionable recommendations, and theoretical and domain depth.Examples include detecting leakage, identifying specific missing baselines, ablations, and metrics, and recognizing mathematical, physical, or hardware constraints.Source: §E.3 · p. 24
  • Across sampled evaluation traces, ScholarPeer was judged superior in 20/20 instances, with no recurring failure patterns relative to the baseline.Source: §E.3 · p. 24

Vs. AI Scientist v2

  • Against AI Scientist v2, ScholarPeer offered stronger literature, experimental, theoretical, and actionable critique, but sometimes missed internal sanity checks and reproducibility or completeness flaws.Reported misses included logical impossibilities, undefined algorithmic operators, and broken manuscript text.Source: Vs. AI Scientist v2 · p. 25

Vs. Agent Review

  • Against Agent Review and the Stanford Agent Reviewer, ScholarPeer better checked timely SOTA baselines, benchmarking standards, experimental integrity, and feasibility, but missed some internal contradictions and theoretical flaws.The comparisons also note occasional misread or miscalculated table values and citations to work published after a review cutoff by the competing reviewer.Source: Vs. Agent Review · pp. 25–26

4. Main Results & Experiments

  • 0.1414 vs. 0.1752 on Synthetic MISE and 1.78 vs. 2.33 vs. 3.30 on IHDP MISE: DBRNet outperformed baselines across almost all reported metrics.The output attributes its approach to disentangled factors and selective bias adjustment; removing re-weighting increased IHDP AMSE by 200%.Source: §4 · p. 27, §3 · p. 27, Ablation & Analysis · p. 27

2. Open Problems & Gaps

  • The domain-history and open-problems outputs trace a shift from binned to continuous and disentangled methods, while identifying high-dimensional confounding, generative stability, and data scarcity as unresolved challenges.The history describes DRNet’s treatment bins, VCNet’s continuity, and later disentanglement and transformer approaches.Source: §1 · p. 27, §2 · p. 28

1. Missing Critical Baselines

  • The DBRNet critique says its evaluation omits key baselines, including SCIGAN, End-to-End Balancing, and DML, and standard benchmarks such as TCGA and MVICU.The critique argues the low-dimensional or synthetic-heavy evaluation may conceal scalability issues and leaves precision-medicine claims insufficiently validated.Source: §1 · p. 28, §2 · p. 28

Part 1: Technical Soundness and Quality

  • The GAFM output describes its GAN and randomized cross-entropy combination as novel for VFL label protection, while noting no formal privacy bound and sensitivity to the loss-weight hyperparameter.It also questions multi-class generalization and omitted non-perturbation defenses; server-hosted generators keep client overhead comparable to vanilla Split Learning.Source: Part 1: Technical Soundness and Quality · p. 29
  • GAFM’s mutual gradient perturbation differs theoretically from Marvell’s external optimized noise by relying on cancellation between conflicting GAN and CE objectives.The passage characterizes this as a shift from adding noise to structural obfuscation.Source: Part 1: Technical Soundness and Quality · p. 29
  • Mean and Median attacks are incremental heuristics within existing clustering frameworks, but provide a stronger benchmark than the standard Norm attack.The passage says they raise the bar for future defense evaluation.Source: Part 1: Technical Soundness and Quality · pp. 29–30
  • The method is limited to binary classification, and its applicability to Vision Transformers and other complex non-CNN architectures remains unproven.The passage notes no ViT-specific evidence or design adaptation, including for attention-related leakage vulnerabilities.Source: Part 1: Technical Soundness and Quality · p. 30

F. Human Evaluation Guidelines

This section documents the guidelines and prompts used to evaluate automated reviews and review assistants. It covers comparisons with human reviews, pairwise assistant rankings, comparative gains and losses, and review criteria including technical accuracy, usefulness, analytical depth, and novelty assessment.

F.1. Objective / F.2. The Evaluation Task

  • Annotators compare two agentic systems with actual human reviews, score each AI review from 1–10 against the best human review, then compare the AI reviews side by side.They first establish the human baseline by noting what reviewers caught and may have missed.Source: §F.1 · p. 30, §F.2 · p. 30, §F.3 · p. 30

F.3. Part I: Individual Scoring (Scale 1-10)

  • Individual scores assess technical accuracy, constructive value, analytical depth, novelty and significance assessment, and overall utility.Technical accuracy checks for factual errors and valid critique; constructive value favors specific actionable suggestions over generic advice.Source: §F.3 · p. 30, §5 · p. 31
  • An AI earns the highest score by uncovering a critical flaw, missing theoretical connection, or vital prior work that all human reviewers missed.Such a review fundamentally improves the critique.Source: §F.3 · p. 31

F.4. Part II: Side-by-Side (SxS) Comparison / F.5. Important Notes

  • Side-by-side judgments select Review 1, Review 2, or a tie based on accuracy, helpfulness, depth, literature positioning, and reviewer preference.Annotators should judge critique validity rather than strictness or presentation, score major hallucinations low, and not infer system identity from blinded slots.Source: §F.4 · pp. 31–32, §F.5 · p. 32

Summarizer Prompt

  • The summarizer prompt requests a concise, accurate, information-dense account of a paper’s contributions, methodology, and results.It is intended as a reliable reference for paper analysis.Source: Summarizer Prompt · p. 32

Literature Reviewer Prompt / Literature Expander Prompts

  • The literature reviewer maps a domain through foundational work, datasets, recent SOTA, competitors, and surveys, while restricting sources to the cutoff date.A literature expander checks for missing foundational, dataset, and temporally intermediate papers and returns unique additions published by that cutoff.Source: Literature Reviewer Prompt · pp. 32–33, Literature Expander Prompts · p. 34

Historian Prompts

  • The historian prompt synthesizes field evolution, recent unsolved problems, and criteria for judging contribution significance at the time.It requests a concise narrative organized into domain history, open problems, and significance criteria.Source: Historian Prompts · pp. 34–35

Baseline Scout Prompts / Novelty Aspect Prompts / Review Generator Prompts

H. Evaluation Prompts

H.1. H-Max Score

H-Max Score Estimation Prompts

  • H-Max evaluations assess technical accuracy, constructive value, analytical depth, and novelty and significance, with external search reserved for novelty claims and restricted to the paper’s cutoff date.Evaluators must support judgments with evidence and direct quotations, using external sources only for novelty and significance.Source: H-Max Score Estimation Prompts · pp. 39–40
  • The H-Max protocol scores each AI review against the best human review for each aspect, using that human review as the score-5 baseline and judging AI value-add.The rubric ranges from verdict-changing insights to failures such as superficial feedback or critical errors and hallucinations.Source: H-Max Score Estimation Prompts · pp. 40–41

H.2. Side-by-Side (SxS) Evaluation

  • The SxS protocol ranks two assistants’ reviews by aspect and overall using evidence-based comparisons, with Google Search limited to novelty and significance assessment.It requires direct quotations, specific examples and counterexamples, and reasoning; novelty searches use only information available by the paper’s cutoff date.Source: §H.2 · p. 41, SxS Evaluation Prompts · pp. 41–43

Summary of Gains Prompts

  • The summary-of-gains prompt clusters recurring evaluation patterns into categories and distinguishes each model’s gains from its losses, reporting up to five factors per category.It asks for a high-level qualitative account of why one model outperforms another.Source: Summary of Gains Prompts · pp. 43–44

H.3. Summary of Gains Analysis

The section uses SxS reasoning traces to synthesize qualitative insights.

H.3. Summary of Gains Analysis

  • SxS reasoning traces are used to synthesize qualitative insights.Source: §H.3 · p. 43
Loading 2601.22638v2…