Guiding LLM Peer Reviewers: The Impact of Score Anchors on Review Evidence and Accuracy

• Prior REF-style studies found correlations between LLM scores and expert or REF-based quality proxies, but these do not establish accurate scores on the target scale.REF-style assessment evaluates published outputs against criteria such as originality, significance, and rigour, linking the score to a rationale that justifies it with strengths and limitations.Source: §1 · p. 1
• The study compares no-guidance reviews with oracle guidance using 98 anonymised UK Unit of Assessment 3 outputs and rounded mean human reference scores on a 1–4 scale.A full guide-score sweep over all score options tests direct score-following.Source: §1 · p. 1
• Underestimates were more often corrected when guidance increased strength framing and reduced weakness framing, while overestimates were more often corrected when guidance reduced novelty/world-leading framing.Source: §1 · p. 2
• The paper introduces a protocol to distinguish direct score copying from rationale changes and a point-level framework for comparing human and LLM evidence used to support score increases or decreases.Source: §1 · p. 2

2 Methods

• Score analysis measured direct score-following and compared no-guidance with oracle-guided scores using exact-match accuracy and mean absolute error against human reference scores.Scores below, equal to, or above the reference were classified as underestimates, correct matches, or overestimates; a mismatch denotes disagreement with the human reference, not an objectively incorrect judgment.Source: §2 · p. 2
• Review-content analysis retained three configurations after screening baseline outputs, then paired each no-guidance review with its corresponding oracle-guided review for the same paper and generation setting.Pairs were grouped by corrected or persistent underestimate or overestimate.Source: §2 · p. 2
• Review text was analyzed using zero-shot sentence-level labels for broad evaluative framing and extracted evaluation points labeled by theme, stance, and score implication.These point labels correspond to evidence type, evaluative direction, and score direction, respectively.Source: §2 · p. 2
• Human-point coverage recorded whether each human-review point was covered, partially covered, or not covered in the corresponding LLM review, encoded as 1, 0.5, and 0.Source: §2 · p. 2

3.1 Settings

• The study tested three paper-input variants across three LLMs, using multiple prompts for Llama-3.1-8B and Falcon3-10B but v4 alone for Llama-3.1-405B.Primary text-level analysis used Llama-3.1-8B v4 with full-text input and three generation settings; other model and prompt settings served as robustness checks.Source: §3.1 · p. 2

3.2 Implementation details

• Qwen3-32B, from a different model family than the reviewed LLMs, judged extracted evaluation points and cross-text coverage, while DeBERTa-v3-large classified sentence-level review frames.Generations were capped at 1024 new tokens.Source: §3.2 · p. 2

3.3.1 Oracle guidance changed scores without simple copying

• Oracle guidance improved exact-match accuracy and reduced MAE in every selected full-text configuration.The largest accuracy gains were reported for Llama-405B v4, Llama-8B v7, and Falcon-10B v7.Source: Table 1 · p. 3, §3.3.1 · p. 3
• Only 31.89% of final scores exactly matched the supplied guide score, and their correlation was 0.32, indicating guidance acted as an anchor rather than a deterministic constraint.No paper-configuration pair copied the supplied score across all four guide-score values.Source: §3.3.1 · p. 3
• Guidance corrected 31 of 68 baseline underestimates and 38 of 82 overestimates, while 35 underestimates and 37 overestimates remained mismatched in the same direction.Underestimates and overestimates are defined relative to the rounded human reference score.Source: Table 2 · p. 3, §3.3.1 · p. 3

3.3.2 Human and LLM reviews used different evidence frames

• Human and LLM reviews emphasized similar themes for higher scores, but LLMs gave less emphasis to international significance and used different themes for downgrade evidence.Human downgrade points more often raised practical-impact and international-significance limits; LLM downgrade points more often concerned generalisability, methodology/rigour, and evidence scale.Source: Table 3 · p. 3, §3.3.2 · p. 3

3.3.3 Guidance steered review frames unevenly

• Corrected underestimates shifted toward a more positive review frame, while corrected overestimates more clearly shifted away from positive framing toward methodological weakness.Persistent mismatches showed weaker movement for underestimates and retained or slightly increased novelty/world-leading language for overestimates.Source: §3.3.3 · pp. 3–4, Table 4 · p. 4
• LLM reviews covered human upgrade/strength points more reliably than downgrade/weakness points, with impact-related and methodology/rigour downgrade evidence weakly covered across overestimated cases.Evidence-scale upgrade points were less covered in persistent than corrected underestimates; additional model and prompt settings supported the same broad patterns of larger frame shifts for corrected mismatches and stronger upgrade-point coverage.Source: Table 5 · p. 4, §3.3.3 · p. 4

4 Discussion

• Score guidance can influence both final scores and review rationales in corrected cases, rather than merely making models copy the supplied score.The steering effect was especially visible when guidance successfully raised baseline underestimates and the review frame shifted positively.Source: §4 · p. 4
• Baseline underestimates were more readily corrected with positive reframing, while overestimates often persisted when reviews retained positive framing or omitted human downgrade evidence.The discussion suggests upward correction may fit instruction-tuned LLMs’ positively framed style, whereas downward correction requires more specific downgrade evidence that was less consistently covered.Source: §4 · p. 4
• Score guidance is useful but incomplete, and systems should be evaluated on whether rationales shift in the direction required by the target score, not only on final-score correctness.Expert downgrade evidence remains harder to incorporate reliably.Source: §4 · p. 4

Limitations

• The 98 self-selected health-domain outputs were REF-style candidates concentrated around 3*, limiting generalization to other settings and the full 1–4 score range.The dataset came from internal assessment rather than random sampling and included relatively few very low-quality outputs.Source: Limitations · p. 5
• Oracle guidance used rounded human reference scores, testing the effect of a correct signal but not the uncertainty of predicted scores in deployment.Source: Limitations · p. 5
• Text-level comparisons relied on automatic sentence labeling and Qwen-based evidence extraction and coverage judgments, which future work should validate against expert-coded evidence points.Source: Limitations · p. 5

A.1.1 Dataset and preprocessing

• The outputs came from a non-random internal REF-style assessment and mainly represent research considered plausible for evaluation, not the full quality range of published work.The assessment reports and scores were private internal materials that are difficult to obtain.Source: §A.1.1 · p. 6
• Each output was assessed by two to four reviewers, and their comments and recommendations informed an adjudicated reference score on a 1–4 scale.Reviewers were intended to assess the output content rather than author, institution, journal, or venue information.Source: §A.1.1 · p. 6
• The study used 98 anonymised health-domain outputs with human reports and adjudicated scores, manually removing author-identifying and publication information where possible.The submitted articles were downloaded as PDFs; anonymisation was intended to reduce reliance on author, institution, or venue signals.Source: §A.1.1 · p. 6
• PDFMiner extracted full text, after which files were manually checked and cleaned of junk text from formulae, images, and tables.All outputs and review reports were in English.Source: §A.1.1 · p. 6
• GROBID section information was post-processed with custom scripts to merge subsections into top-level sections using textual cues.These top-level sections formed the LLM inputs.Source: §A.1.1 · p. 6
• The study used title-and-abstract, title/introduction/conclusion, and full-text input variants; the main text-level analyses used full text.The broader output set also included the two shorter input conditions.Source: §A.1.1 · p. 6

A.1.2 Prompt templates

• Three prompt variants differed only in block order: v4 used rule–format–paper text, v7 paper text–rules–format, and v9 rules–paper text–format.The same system prompt was used across conditions; baseline reviews used the no-guidance task prompt, while oracle-guidance prompts included an external score block.Source: §A.1.2 · p. 6

A.1.3 Model, decoding, and configuration screening

• Outputs used Llama-3.1-8B-Instruct, Falcon3-10B-Instruct, and Llama-3.1-405B-Instruct; the first two used prompts v4, v7, and v9, while the 405B model used only v4.The 405B prompt set was limited because of generation cost.Source: §A.1.3 · p. 6
• The candidate set crossed three input-length conditions with 11 decoding settings, yielding 33 input–decoding configurations per model/prompt setting.Inputs ranged from title and abstract to full text; decoding included greedy and sampling across five temperatures and two top-p values.Source: §A.1.3 · p. 6
• Screening compared configurations’ signed score-difference vectors using normalized mutual information, preferring lower shared error information alongside acceptable score quality.Each vector recorded the LLM score minus the rounded human reference score for each paper, addressing correlated errors across outputs.Source: §A.1.3 · p. 6
• Three full-text decoding settings were retained for the main analysis: temperature/top-p 0.1/0.9, 0.3/0.8, and 0.5/0.9.Full-text input was retained for its richer paper evidence and because it was consistently among the stronger candidates; Llama-3.1-8B prompt v4 was primary, while other model and prompt settings served as robustness checks.Source: §A.1.3 · p. 7, Table A1 · p. 7

A.1.4 Annotation labels, prompts, and text features

• Text features used sentence labels, extracted evaluation points, and coverage judgments; sentence and point features were calculated separately, with guided-minus-baseline deltas where relevant.Theme distributions were computed within source and score-implication groups, while human-point coverage used human evaluation points as the reference.Source: §A.1.4 · pp. 7–8
• Sentence-level zero-shot labels classify review sentences by evaluative stance and by themes such as significance, contribution, impact, evidence, and methodological or contextual limitations.The stance labels distinguish strength, weakness, mixed evaluation, and neutral description.Source: §A.1.4 · p. 8
• Qwen3-32B extracted human and LLM review points using a shared schema, then judged whether human points were covered; stance and score implication were treated separately.Coverage was judged as covered, partially covered, or not covered, using human evaluation points as the reference.Source: §A.1.4 · p. 8

A.1.5 Compute resources and runtime

• The 8B and 10B settings required about 18.3 sequential hours per model/prompt/input setting, while Llama-3.1-405B used 165 files and about 412.5 sequential GPU-job hours.Each 8B/10B model/prompt/input setting generated 55 files, yielding 495 files for each of Llama-3.1-8B and Falcon3-10B across three prompt variants and three input lengths.Source: §A.1.5 · p. 9

A.2.1 Supplementary score-guidance results

• Supplementary score analyses extend selected full-text results across input lengths and decoding settings, and separately test whether full-sweep guidance determines final scores.The broader results compare no-guidance outputs with the oracle-guided subset where the supplied score equals the rounded human reference score.Source: §A.2.1 · pp. 9–10, Table A4 · p. 12, Table A5 · p. 12

A.2.2 Score transition results

• For each selected model, prompt, and decoding setting, score transitions track baseline mismatches corrected, persistent same-direction errors, and initially correct cases that became mismatched across 98 papers.Baseline and oracle-guided scores are classified as underestimates, correct matches, or overestimates relative to the rounded human reference score.Source: §A.2.2 · p. 10, Table A6 · p. 13

A.2.3 Summary of robustness checks across settings

• Score-level effects were most stable across selected settings; text-level effects were broadly consistent, especially links between scores and review frames, while human downgrade-evidence coverage was more model-dependent.The robustness checks assess whether the direction of findings holds across settings, not whether every model produces identical numerical effects.Source: §A.2.3 · p. 10, Table A7 · p. 13