PaperlayerView exact revision on arXiv

Source-linked AI summary

Local AI pre-screening for human triple-blind peer review in health sciences

Rodrigo Martins Boos

arXiv:2608.14625v1cs.CYcs.AIcs.DL

TL;DR

Peer review faces growing submission volume and limited expert attention. This paper proposes disclosed, triple-blind, locally hosted multi-LLM pre-screening with mandatory human review, but reports no empirical validation yet.

  • Problem

    Peer review faces growing submission volume and limited expert attention, including analogous pressure in health sciences journals.

  • Method

    The paper proposes a disclosed, auditable, triple-blind workflow using three locally hosted LLMs for pre-screening before mandatory human review and editorial decision-making.

  • Results

    The proposed architecture has not yet been piloted on real manuscripts, so its accuracy, error rates, and time savings remain unmeasured.

  • Takeaways & Limitations

    The design offers a governed, human-supervised template that other journals facing similar pressure can adapt.

  • Takeaways & Limitations

    The architecture lacks empirical validation, and its proposed rubric thresholds require calibration against pilot comparisons with human and editorial decisions.

Abstract

from arXiv · show

Academic peer review is under mounting strain: NeurIPS 2025 received 21,575 submissions, ICLR 2025 received 11,603, and ICML 2025 received 12,107. This volume has outpaced the supply of qualified reviewers, and large language models (LLMs) are already filling the gap, largely undisclosed. An independent analysis of ICLR 2026 found roughly 21% of its 75,800 peer reviews were fully AI-generated, with over half showing some AI involvement (up from 15.8% in 2024). Documented risks include hallucinated citations in accepted papers and hidden prompt-injection instructions embedded in manuscripts to manipulate AI reviewers into favorable assessments. We propose a triple-blind, multi-LLM pre-screening framework for peer review, developed for a health sciences journal, that formalizes and discloses AI involvement while preserving human reviewers as the final decision-making authority. The framework routes a submission through five stages -- sanitization/anonymization, parallel AI pre-screening, an automated check gate, blinded human review, and editorial adjudication -- with return-to-author loops at the check and editor stages. Addressing the confidentiality concerns behind NIH/NSF bans on submitting unpublished proposals to third-party generative AI, all three AI reviewers run on locally-hosted, open-weight LLMs, keeping manuscript content within the journal infrastructure. The closest precedent, Shen et al., benchmarked five open-source LLMs on quartile classification of 200 manuscripts and found accuracy insufficient (35% exact-match) for autonomous use, supporting our decision to retain mandatory human adjudication. This transparent, human-supervised design offers a defensible alternative to today's opaque, unregulated AI use in peer review, potentially reducing the substantial delay of traditional review (avg. 13 weeks to first decision) without displacing human judgment.

1 Introduction

Peer review is buckling under submission growth while undisclosed LLM use is already widespread and associated with measurable reliability concerns. The paper proposes a transparent, auditable, triple-blind, multi-LLM pre-screening process that keeps humans in control.

  • Submission pressure: 24,371 submissions had already reached ICML 2026, while NeurIPS 2025 received 21,575 and ICLR 2025 received 11,603.Health sciences journals face an analogous, though less publicly quantified, pressure.
  • Undisclosed AI use: 21% of 75,800 ICLR 2026 reviews were entirely AI-generated, and more than half showed some AI involvement.The entirely AI-generated share increased from 15.8% at ICLR 2024.
  • Reliability concerns: Over 100 hallucinated citations appeared across 51 of 4,841 papers accepted at NeurIPS 2025.This finding illustrates a measurable consequence of AI-assisted review and assessment.
  • Reliability concerns: GPT-4 detected deliberately inserted errors in 7 of 13 short computer science papers, indicating useful but unreliable review capability.The paper uses this mixed evidence to support a structured, human-supervised role for LLMs.
  • Proposed framework: The proposed workflow brings covert, large-scale AI-assisted review into a disclosed, auditable, triple-blind, multi-LLM process with human supervision.Existing screening tools do not perform general scientific merit assessment with multiple independent generalist LLMs in a triple-blind architecture.

2 Related Work — Differentiation from Shen et al. (2026)

Shen et al. evaluated five open-source LLMs on quartile classification of 200 transplantation manuscripts, a narrower task than substantive merit review. Their 35% exact-match accuracy supports using LLMs for pre-screening rather than replacing human peer reviewers.

  • Scope of Shen et al.: Shen et al. evaluated five open-source LLMs on 200 transplantation manuscripts using four prompting strategies for single-pass journal-quality quartile classification.The prompting strategies were zero-shot, few-shot, tree-of-thoughts, and retrieval-augmented generation (RAG).
  • Findings from Shen et al.: 35% exact-match accuracy was achieved by the best-performing configuration.The task predicted which journal-quality quartile each manuscript belonged to.
  • Implication for the proposed framework: Shen et al. concluded that current open-source LLMs were not reliable enough to replace human peer reviewers, supporting mandatory human escalation after AI pre-screening.The study found no significant affiliation bias.

3 System Architecture and Design

The framework processes each manuscript through five sequential stages. It includes return-to-author loops after the automated check and editorial review stages.

  • The framework processes a manuscript through five sequential stages.
  • A return-to-author loop follows the automated check at stage 3.
  • A second return-to-author loop follows editorial review at stage 5.

1. Sanitization & Anonymization (automated, local) … 3.4 LLM pre-screening rubric

The framework sanitizes and anonymizes manuscripts locally before three independent LLMs apply a structured rubric, while preserving triple-blind human review and editorial adjudication. Automated screening routes submissions through a check gate, with author-return loops and identity disclosure withheld until editorial approval.

  • 2. AI Review — three independent LLMs score the manuscript against a shared rubric: The automated gate sends passed manuscripts to three human reviewers blinded to author identity and AI assessments, while failed or ambiguous cases return to the author.The editor later reviews both AI and human assessments and returns unapproved manuscripts with consolidated observations from prior stages.
  • 2. AI Review — three independent LLMs score the manuscript against a shared rubric: The process extends classical triple-blind review by placing a disclosed, auditable AI layer before human review and delaying author-identity access until both layers conclude.The architecture comprises sanitization and anonymization, AI pre-screening, an automated check gate, blind human review, and editorial adjudication with return-to-author loops.
  • 3.2 Sanitization protocol: Before review, automated sanitization strips hidden text, clears document metadata, flags mismatched non-visible layers, and removes embedded executable content.The pipeline logs sanitization actions and retains the log for audit without showing it to reviewers.
  • 3.2 Sanitization protocol: The system produces a canonical plain-text/Markdown manuscript and an auditable sanitization log using a custom Node.js extraction and stripping pipeline.Implementation uses pdf-parse, mammoth, and regex/Unicode-category stripping rather than a third-party sanitization SaaS.
  • 3.3 Anonymization protocol: Anonymization removes names, affiliations, acknowledgments, email addresses, and ORCID identifiers, while self-citations become generic markers and distinctive narratives receive manual editor review.Self-citations are flagged rather than removed so citation practice remains assessable without exposing author identity.
  • 3.3 Anonymization protocol: The original manuscript remains encrypted and access-controlled within the anonymization system, locked through stages 2–5 and unlocked only upon editorial approval.Reaching stage 5 or passing the AI and human layers does not itself reveal author identity.
  • 3.4 LLM pre-screening rubric: A submission passes to human review when the three-LLM mean is ≥3.5 across all criteria and no single criterion is below 2; otherwise, it returns to the author with merged feedback.Each LLM also supplies structured chain-of-thought justification for every criterion before assigning numeric scores.

3.5 Prompt versioning & reproducibility … 3.9 Ethical disclosure

The framework preserves reproducibility, defends against prompt injection, and lets authors preselect escalation policies while keeping human reviewers and editors responsible for substantive review and publication decisions. Local open-weight infrastructure keeps manuscripts within journal systems, and AI use is disclosed before submission.

  • 3.5 Prompt versioning & reproducibility: Each review cycle permanently stores the exact prompt and rubric version, model checkpoint, sampling parameters, and execution timestamp.These records address reproducibility risks from silent provider-side model updates between reviews.
  • 3.6 Anti-prompt-injection safeguard (defense in depth): Sanitization removes known injection vectors, while prompts require LLMs to treat manuscripts as data and flag embedded instructions rather than obey them.Any flag automatically fails the Check gate and routes to manual editor inspection regardless of scores.
  • 3.7 LLM →human escalation criteria, and the author-authorized rejection mode: Authors choose escalation policy at submission: “Always human review” is the default, and LLMs cannot issue terminal rejections.Every nonwithdrawn manuscript eventually reaches a human reviewer or editor under this policy.
  • 3.7 LLM →human escalation criteria, and the author-authorized rejection mode: Under the opt-in AI-authorized rejection policy, two of three LLMs must independently score below 3.5 on the same criterion.A triggered manuscript returns to the author with specific low-scoring criteria and private LLM justification, without human review.
  • 3.7 LLM →human escalation criteria, and the author-authorized rejection mode: Human reviewers receive only the anonymized manuscript and know it cleared AI pre-screening, never the AI scores, justifications, or threshold.The editor alone sees both AI scores and human assessments, and de-anonymizes only after approving the manuscript.
  • 3.8 Infrastructure & data governance: All three LLM reviewers run on locally-hosted, open-weight infrastructure, ensuring manuscript content never leaves the journal’s infrastructure during pre-screening.Candidate models include Gemma 3 27B and Qwen3 14B/32B, feasible on consumer-GPU or small multi-GPU setups.
  • 3.9 Ethical disclosure: Authors are informed before submission that local LLMs process manuscripts, while substantive peer-review reports and publication decisions remain exclusively human acts.This disclosure follows ICMJE guidance and COPE’s position on transparent AI use in review.

4 Implications, Ethical Positioning, and Open Questions

The proposal positions AI as disclosed, rubric-scored triage upstream of exclusively human peer-review reports and decisions, while treating AI-authorized rejection as an opt-in boundary case requiring further safeguards. It targets reviewer-recruitment delays rather than replacing human judgment, and is situated alongside emerging evidence that disclosed AI-assisted review can operate at institutional scale.

  • Ethical positioning: AI output is a logged, labeled triage signal kept structurally and visually separate from human review, rather than a peer-review report or editorial assessment.The framework uses role separation to address COPE’s prohibition on undisclosed generative-AI production of reviews, assessments, or decisions.
  • Ethical positioning: Human reviewers alone author peer-review reports, and the human editor alone makes publication decisions; AI output appears in neither artifact.The AI layer operates strictly upstream of both human-authored artifacts.
  • AI-authorized rejection: 40.6% of 266 desk-rejection letters cited only generic criteria, while 18.8% cited only a generic strict-evaluation process, together approaching 60% without actionable reasons.The opt-in AI-authorized rejection mode is framed as isolating early diagnostic value from socially witnessed rejection, but an unaudited AI-only path could silently reject legitimate work.
  • Operational implications: 3.6 reviewer invitations per accepted review was projected for 2025, up from 2.7 in 2017, as more than half of invitations were already declined.The architecture returns manuscripts scored not yet ready before the multi-week reviewer-recruitment funnel; AI processing takes seconds to low minutes per manuscript.
  • Related evidence: 22,977 main-track papers in the AAAI-26 pilot each received one flagged AI review in under a day, at below $1 per paper, with average recall gain +0.21 over baseline.The reported recall result had p = 6.1 × 10−30, supporting institutional feasibility of disclosed AI-assisted review.

5 Limitations

The framework remains unvalidated and leaves important questions about model bias, security, scalability, ethics, and regulatory status unresolved. Its confidentiality-first design deliberately accepts capability and operational trade-offs while retaining human oversight.

  • Empirical validation: The architecture has not been piloted on real manuscripts, so its thresholds and projected accuracy, error rates, and time savings remain unmeasured.The proposed working values include mean score ≥3.5 and no criterion below 2; validation requires comparison with final human and editorial decisions.
  • Model bias and instability: LLM assessments can be less variable, more positive, and overconfident than human reviews, while three models cannot eliminate correlated bias or shared blind spots.Comparable open-weight models may train on overlapping data, limiting the protection provided by model plurality.
  • Model capability: Confidentiality requires locally hosted open-weight reviewers in the 14B–32B parameter range, but these remain behind frontier commercial models on subtle reasoning tasks.The trade-off is deliberate: eliminating third-party data exposure takes priority over frontier-model capability.
  • Security: Sanitization addresses known prompt-injection vectors but is not sufficient against evolving adversarial techniques, and treating manuscript content as data is not a guarantee.Documented attacks have included white-on-white text, but future techniques may be more sophisticated.
  • Scale and cost: Compute cost and throughput for running three independent local models at full journal scale were not evaluated and are deferred to the pilot phase.Unlike the AAAI-26 pilot, this framework reports no concrete per-paper cost or processing-time figures.
  • Ethics and regulation: AI-authorized rejection remains ethically novel and untested, while the framework’s alignment with NIH/NSF confidentiality rationale and COPE norms is voluntary rather than binding.The rejection mode depends on author autonomy and genuinely informed consent, and the regulatory alignment should not be overstated as endorsement.

6 Conclusion

The paper frames AI involvement in peer review as an established but insufficiently governed reality and proposes disclosed, human-supervised triple-blind pre-screening. It presents the architecture as a proposal requiring preparatory bench validation and a live three-arm pilot before its value can be assessed.

  • Conclusion: Roughly one in five reviews at a major 2026 machine learning conference were AI-generated, while over a hundred hallucinated citations reached an accepted proceedings.The conclusion argues that the central issue is governing and disclosing AI involvement rather than deciding whether AI will affect peer review.
  • Conclusion: Three independent, locally-hosted LLMs would conduct structured pre-screening before mandatory human peer review and editorial decisions in a triple-blind architecture.The review report and publication decision remain exclusively human, while the design aims to reduce how many manuscripts enter the sl…
  • Validation: Preparatory bench validation would run sanitization, AI review, and the Check gate on labeled accepted, rejected, and prompt-injection-seeded manuscripts against known ground truth.This proposed test evaluates the whole pipeline rather than a single classification task.
  • Validation: A live three-arm pilot would compare AI-only, hybrid, and conventional human-only review on decision quality, time-to-decision, and psychological burden.The architecture’s value proposition would need revisiting if hybrid review fails to outperform AI-only on decision quality or reduce time-to-decision relative to conventional review.

Footnote

The manuscript reports no generated or analyzed datasets and has not yet been piloted on real manuscripts. It declares no conflicts of interest, provides ethical accountability, and is distributed under CC BY 4.0.

  • Reporting checklist: No specific EQUATOR reporting guideline applies because this architecture proposal reports no clinical trial, systematic review, or empirical dataset.
  • Data sharing: No datasets were generated or analyzed, and the proposed architecture has not yet been piloted on real manuscripts.The manuscript identifies this as a design proposal and refers to piloting limitations.
  • Conflicts of interest: The author completed the ICMJE uniform disclosure form and declared no conflicts of interest.
  • Ethical statement: The authors accept accountability for investigating and resolving questions concerning the accuracy or integrity of any part of the work.
  • Open access: The article is Open Access under the Creative Commons Attribution 4.0 International License, permitting unrestricted redistribution and reuse with proper citation.The license is identified as CC BY 4.0.

Abbreviations

This section defines abbreviations used throughout the paper, spanning artificial intelligence, peer-review organizations and conferences, publishing bodies, document formats, and research technologies.

  • Artificial intelligence and conferences: AAAI, AI, ICLR, ICML, and NeurIPS denote major artificial-intelligence organizations, concepts, and conferences.AAAI is the Association for the Advancement of Artificial Intelligence; AI is artificial intelligence; ICLR, ICML, and NeurIPS expand to their respective conference names.
  • Research and software terminology: API, ORCID, RAG, RCT, and SaaS denote software interfaces, researcher identification, generation methods, trial designs, and service delivery.The abbreviations expand to application programming interface, Open Researcher and Contributor ID, retrieval-augmented generation, randomized controlled trial, and software as a service.
  • Publishing and research organizations: COPE, ICMJE, JMIR, NIH, and NSF refer to publication-ethics, medical-journal, journal, and funding organizations.These abbreviations expand to the Committee on Publication Ethics, International Committee of Medical Journal Editors, Journal of Medical Internet Research, National Institutes of Health, and National Science Foundation.
  • Technical formats and tools: DOCX, GPU, OCR, and PDF denote document, computing, text-recognition, and file-format technologies.DOCX expands to Office Open XML Document; GPU to graphics processing unit; OCR to optical character recognition; and PDF to Portable Document Format.
Loading 2608.14625v1…