Benchmarking locally hosted language models for journal editorial work on a compact desktop workstation

Introduction

• Generative models can introduce detectable defects, including fabricated or erroneous citations and real papers cited for claims they do not support.The text reports that approximately 300 papers contained at least one hallucinated citation, including more than one hundred accepted conference papers [15].Source: Introduction · p. 2
• The editorial bottleneck may shift toward processing more adequate submissions reliably and efficiently as submission volume grows faster than the available reviewer pool.The text describes conflicting evidence on AI's quality effects, while cited studies agree that output increases [17,18].Source: Introduction · p. 2
• Submitted manuscripts may contain proprietary results, and publisher policies restricting transmission to external AI services motivate inference on hardware controlled by the editorial office.The text notes that confidentiality concerns applied to reviewers are also expected to extend to editorial offices handling the same documents.Source: Introduction · pp. 2–3
• The study asks how much editorial work open-weight models can perform, whether capability scales predictably with model size, and how task formulation, tool access, and hardware affect performance.These questions frame a benchmark of locally hosted language models for editorial work.Source: Introduction · p. 3
• Eight tasks drawn from author guidelines, manuscripts with seeded and independently verified defects, and published reviews of a preprint were evaluated on a compact desktop workstation.A short program without a language model provided a deterministic baseline on the same items.Source: Introduction · p. 3

Materials and methods

• T1–T4 tested guideline compliance, review support, reference conversion and rule retrieval, and whether models refused to answer unanswerable questions.T2 scored against 12 substantive points extracted from three published reviews; T4 included unanswerable guideline and manuscript questions, for which inventing an answer was the target failure.Source: Tasks, guideline document,… · pp. 4, 6
• T5–T8 tested figure and table checking, cross-section numerical consistency, category-specific exemptions, and reference verification using Crossref and the NLM Catalog.T5 used four figures, three containing five defects; only three defects strictly required vision, so its score is an upper bound on visual capability. T8 included a non-resolving DOI and two references that disagreed with their resolved records.Source: Tasks, guideline document,… · pp. 4, 6
• The benchmark used a journal’s Instructions for Authors, three study-composed Regular Article manuscripts, six distractors, and three items involving internal scientific inconsistencies; MS-E had five exempt rules.Source: Tasks, guideline document,… · p. 5
• Task-specific item sets included eight references and 10 Japanese rule questions for T3, and eight guideline plus five manuscript questions for T4.T3 tested retrieval from a long English guideline and responses in Japanese; T4 included four answerable and four unanswerable guideline questions, plus five questions about absent manuscript elements.Source: Tasks, guideline document,… · p. 6
• Twenty open-weight models spanning 3.3–81 GB were evaluated, but task participation depended on model capabilities and included an omission from T8.Seventeen models ran text tasks, all 12 with vision encoders ran T5, and 15 of 18 tool-capable models ran T8; the two qwen3-vl variants were omitted from T8 despite supporting tool calls.Source: Models and the tasks each could run · p. 6
• Scoring used task-specific ground truth: T1, T6, and T7 matched anchored findings independently, while T2 matched its distinct reference criticisms one-to-one.T3, T4, and T8 checked expected fields for exact presence; T5 used permissive keyword matching, making its scores an upper bound in that respect.Source: Prompts, inference parameters, and… · p. 8
• T3 and T4 were reported as separate paired measures because their item sets test different abilities and can produce opposing model rankings.For T3, reference conversion and Japanese-question answers diverged; on T4, guideline questions and absent-element checks also diverged for some models.Source: Prompts, inference parameters, and… · pp. 8–9
• 12 of 15 repeated conditions scored identically across three seeds; the other three differed by one violation, leaving reported orderings unchanged.The repeated conditions were selected from models leading each task, and each seed was reported separately rather than averaged.Source: Prompts, inference parameters, and… · pp. 7–9, Table 2 · p. 9
• The no-model baseline used regular expressions and arithmetic on manuscript text; because it did not parse Word or LaTeX structure, its score was a lower bound for scripting.The checker was evaluated on the same manuscripts and under the same conditions as the language models.Source: Prompts, inference parameters, and… · p. 10

24.04.3 (kernel 6.14.0-1013-nvidia) with CUDA 13.0 and driver 580.95.05. Because memory is unified

• The evaluation ran on an NVIDIA DGX Spark with 119 GiB unified CPU/GPU memory, enabling single-machine tests of models from 3.3 to 81 GB.The workstation used a GB10 Grace Blackwell SoC and a 20-core Arm CPU.Source: Workstation and inference stack · p. 10, §24.04.3 · p. 10
• Most models used ollama 0.17.6, while qwen3.8:27b and gemma4 used 0.32.15, an uncontrolled runtime-version factor.The newer models required the newer runtime, while upgrading the system service was not possible.Source: §24.04.3 · p. 10
• 518,067 tokens/s was an uninformative cache-inflated throughput; speed was instead measured after unloading each model on unique passages from published papers.The two-node comparison was a throughput microbenchmark, not one of the eight tasks.Source: Measurement of inference speed and… · pp. 10–11
• Peak execution memory was sampled at five context lengths from 4k to 128k to distinguish model-weight footprint from KV-cache growth.Peak memory was defined as the maximum observed during execution.Source: Measurement of inference speed and… · p. 11
• Every tested condition was reported, including unusable outputs and requests exceeding the equal 3,600-second wall-clock budget.A model's inability to complete within that shared budget was treated as a measurement result, not excluded.Source: Measurement of inference speed and… · p. 11
• Claude Code running claude-opus-5 carried out benchmark construction, execution, analysis, and drafting under the author's direction; the author reviewed its work and corrected intermediate Discussion conclusions.Judgments about seeded defects and unmatched model findings were made by Claude Code and reviewed by H.O.Source: AI Scientist system that carried out… · p. 11
• The measurements themselves came from repository scripts and can be rerun without language-model assistance.The author reviewed and takes responsibility for the system's output and the paper's content.Source: AI Scientist system that carried out… · p. 11

Detection of seeded guideline violations (T1)

• 31/40 violations were detected by the deterministic checker, 36/40 by the strongest model, and all 40 by their separately measured union.The checker took under 0.1 seconds and no memory, with no false positives on the compliant control; the model took 722 seconds and 81 GB and raised one false positive.Source: Detection of seeded guideline… · pp. 11–12, Table 3 · p. 12
• The checker and model found 27 violations in common, while each of the 40 was missed by some model condition, arguing against a single-model design.Relative to the checker, the model added nine detections at approximately 7,000 times the wall-clock time and with an 81 GB footprint.Source: Detection of seeded guideline… · p. 12
• All 62 model conditions missed the table-numbering-order violation, which the deterministic checker detected by comparing citation order with numbering.For this item, the deterministic layer was the only successful route observed.Source: Detection of seeded guideline… · p. 12, Figure 3 · p. 14
• Task completion showed a bimodal pattern: 12 of 62 conditions detected no violation at all.Three exhausted the 24,576-token generation budget and returned no answer, two exceeded the time budget, five yielded no recoverable findings, and two emitted barely a hundred tokens.Source: Detection of seeded guideline… · p. 12
• Four of the 40 violations appeared to require semantic interpretation; the other nine beyond the checker’s reach included layout, lookup, and section-placement cases.Of those nine, four required judging wording or what a statement covers, three were layout properties, one was an authority-file lookup, and one was a section-placement rule.Source: Detection of seeded guideline… · p. 14

Effect of model weight size on score (T1–T8)

• Across best tested configurations, Spearman coefficients ranged from −0.19 to +0.13, with no consistent monotonic size-score ordering on any task.The analysis used each model’s highest score, which favors models tested in more configurations; no significance test was reported, so lack of detectable association is not evidence of none.Source: Effect of model weight size on score… · pp. 14–16, Figure 4 · p. 15
• On T1, the 81 GB model detected 36/40 violations, but a 17 GB model detected 33; within the gpt-oss family, the 65 GB model scored 16/40 versus 19/40 for its 13 GB sibling.For figure checking, the four perfect scores were at 17, 17, 23, and 38 GB; five models at 62–81 GB scored 3 or 4, and a 67 GB model had the largest false-positive count.Source: Effect of model weight size on score… · p. 15

Model rank within each task (T1–T8)

• No model led across measured tasks: models leading on guideline compliance or peer-review support could fail journal-abbreviation lookups, while another returned all four abbreviations but ranked 10th on compliance.Figure 5 shows within-task rank trajectories; the observed rank variation suggests task-specific selection may help, though generalization is limited by the single T2 preprint and five T5 defects.Source: Model rank within each task (T1–T8) · pp. 16–17, Figure 5 · p. 16
• Task-specific roles selected an 81 GB model for T1, 17 GB models for T2 and T5, and a 9.6 GB model for T8; together they need 136 GiB, beyond the machine.Loaded one at a time, the largest requires 84.8 GiB and fits; no single model filled all four roles.Source: Model rank within each task (T1–T8) · p. 16

Effects of prompt structure, quantization, and model generation on score (T1)

• On T1, among 14 models measured under both prompt forms, the checklist yielded more detected violations for 9 models, though it did not move every model upward.For qwen3.8:27b, gains were on single-location violations, while the harder class did not improve; qwen3.6:27b gained on the harder class. One comparison detected eight additional violations while occupying 11 GiB less.Source: Effects of prompt structure,… · p. 17
• In T8, 6 of 15 tool-capable models returned all four correct abbreviations, but the other 9 failed, including six that made 2–19 tool calls and still answered wrongly.Only 17 of 62 T1 conditions reported the PNAS abbreviation without a tool; successful lookup required both invoking the tool and using its returned authority-file information.Source: Effects of prompt structure,… · p. 18
• A 14 GB model produced 4/40 after 37 minutes of repetitive free-form output, but checklist structure yielded 19/40 in 156 seconds and area-split prompting yielded 22/40 without looping.Other models either completed T1 only under structure or collapsed under it, so one prompt format did not assess every model reliably.Source: Effects of prompt structure,… · p. 18

Peak memory and the prefill–decode split (T1 and T2)

• At 128k context, three models exceeded 96 GiB and were least usable; totals were 17.2 GiB for a 17 GB model, 113.8 GiB for a 67 GB model, and 84.8 GiB for the 81 GB T1 leader.The respective KV-cache additions were +1.1 GiB over the 17 GB model’s 4k footprint, +48.8 GiB over the same range, and +4.4 GiB.Source: Peak memory and the prefill–decode… · p. 19

• 1.5–1.9% decode gains for the 15.65 GiB model and 3.5–17.5% for the 63.06 GiB model were smaller than the prefill gains.For the 63.06 GiB model, two-node decode ran at 24.8–25.0 tokens/s versus 21.3–24.0 tokens/s on one node; decode is sequential, limiting parallelization.Source: Prefill and decode throughput on one… · p. 20
• 0.46–0.54× single-node throughput over TCP made two-node splitting slower, while RDMA-to-TCP throughput was 3.3–3.8× at the two tested prompt lengths.The comparison was between RDMA and TCP on this inference stack, not distributed inference in general.Source: Prefill and decode throughput on one… · p. 20
• 6–8 s, or about 1%, was saved by the prefill speedup in a 605 s T1 run with 16.9 s of prefill, so scaling out is most likely to help long-input-prefill workloads.The phase split does not identify the binding hardware constraint: bandwidth was not varied, and whether the relationship holds on hardware with several times the bandwidth cannot be established from this single platform.Source: Peak memory and the prefill–decode… · p. 19, Prefill and decode throughput on one… · p. 20
• 1.5–1.8× higher prefill throughput came from two-node RDMA splitting across prompt lengths and model sizes, including a 1.54–1.65× gain for the 15.65 GiB model.For the 63.06 GiB model, the gain was 1.64–1.77× across five prompt lengths; the authors attribute the gains to compute parallelism, not memory relief.Source: Figure 8 · p. 20, Prefill and decode throughput on one… · p. 20

Discussion

• The strongest model detected 36 of 40 seeded guideline violations, the 17 GB model detected 33, and the best peer-review model recovered 6 of 12 reference points in 315 s.Precision against unsupported criticism was not measured; 9 of 17 models detected the point one reviewer considered most important, while 8 did not, including 4 with no usable output.Source: Discussion · pp. 21–22
• Across the best configurations, weight size had no consistent monotonic association with score on any of eight tasks; layer assignment, model generation, and task phrasing mattered, while raising quantization did not help.Prompt structure substantially altered scores within individual models.Source: Discussion · p. 21
• The deterministic checker recovered three quarters of seeded violations in under a tenth of a second, including four missed by the top model, supporting a workflow of checks, lookup tools, models, then humans.Without the deterministic baseline, the models’ contribution would have appeared to be 36 recovered violations rather than nine; deterministic checks should precede inference, with models reserved for semantic judgments and humans for unresolved cases.Source: Discussion · p. 21
• Capacity-insensitive failures included a nearly five-order inconsistency found by only 1 of 17 models and an isotropy contradiction found by none; journal-name lookup failures were resolved when models used a bibliographic tool.Nine of 17 models converted every journal name correctly, including failures by the largest model; the unit-conversion inconsistency was found by a 38 GB model and missed by the other 16.Source: Discussion · p. 21
• A 17 GB model occupied 17.2 GiB and the strongest compliance model 84.8 GiB, but keeping the best model for every task resident required 136 GiB on a 119 GiB machine.For compliance conditions producing more than 5,000 output tokens, decode accounted for 94–100% of wall time in 55 of 88 conditions; long-input review shifted more time to prefill, though decode remained the majority in two of four measured conditions.Source: Discussion · p. 22
• At a 4k prompt, a 42 GB dense model decoded at 4.6 tokens/s versus 43.3 tokens/s for a same-generation mixture-of-experts model with 3B active parameters; the gap narrowed to 1.3-fold at 65k tokens.Adding a second node raised prefill throughput by 1.5–1.8x; decode rose 1.5–1.9% for a 15.65 GiB model and 3.5–17.5% for a 63.06 GiB model. Bandwidth was not varied independently.Source: Discussion · p. 22
• The benchmark used the authors’ manuscripts because real submissions were unavailable, so absolute recall may not represent field performance, although within-benchmark comparisons remain informative.The peer-review test used one preprint from one subfield; most conditions had one random seed, repeats covered only 15 conditions, and the figure task did not isolate visual reasoning because only three of five defects strictly required image information.Source: Discussion · pp. 22–23
• The top-scoring model ran on a newer runtime version, and throughput measurements came from one runtime family, leaving a potential confound and software-stack-specific results.A single-runtime replication and tests with alternative runtimes would address these limitations.Source: Discussion · p. 23
• Two-node tests measured prefill and decode microbenchmarks rather than the full task suite; the end-to-end estimate was calculated from measured phase throughputs.Source: Discussion · p. 23
• One prompt variant was evaluated on only 7 of 17 models because each condition required eight inference calls, though no principal conclusion depended on it.Completing the comparison would provide a stronger test of prompt-structure effects.Source: Discussion · p. 23
• The deterministic layer operates on plain text without parsing document structure, so a production implementation using submitted-file formats could recover violations assigned to the model.The authors expect this to narrow the model’s incremental contribution and identify it as a tractable extension.Source: Discussion · p. 23

Conclusion

• Across the best configurations tested, capability did not increase consistently with model size.In guideline-compliance screening, the checker used no model and ran as a deterministic plain-text check.Source: Conclusion · p. 23
• The deterministic checker detected 31 of 40 seeded violations and the highest-scoring model detected 36, while their combined detections covered all 40.Of the nine violations the checker could not decide, four required semantic interpretation; the others involved document structure, lookup, or placement rules.Source: Conclusion · p. 23
• The best model recovered 6 of 12 points raised by human reviewers in the peer-review task.The review points were drawn from three published reviews.Source: Conclusion · p. 23
• The conclusion recommends deterministic checks and external tools for reliably solvable tasks, reserving language models for work requiring semantic interpretation.This division of labor retains the confidentiality benefits of local deployment.Source: Conclusion · p. 23

Author team composed of AI scientist(s) and human scientist(s)

• Haruka Ozaki is listed with affiliations at RIKEN Center for Biosystems Dynamics Research and the University of Tsukuba.The author line also names Claude Code and Claude Opus 5.Source: Author team composed of AI… · p. 24
• The listed affiliations are RIKEN Center for Biosystems Dynamics Research and the University of Tsukuba.Both affiliations appear alongside Haruka Ozaki's name.Source: Author team composed of AI… · p. 24
• The author metadata reports Claude Code version 2.1.238 and model claude-opus-5.Source: Author team composed of AI… · p. 24

Data availability

• The listed materials include the benchmark, execution harness, ground-truth data, all 313 measured conditions, figure-generation code, and a chronological working log.Source: Data availability · p. 24
• Tabulated numbers are generated from stored results rather than transcribed by hand, and ground truth passed 52 independent arithmetic checks before model scoring.Source: Data availability · p. 24