PaperlayerView exact revision on arXiv

Source-linked AI summary

Science or Slop?: Benchmarking and Mitigating Scientific Slop in AI-Generated Papers

Yerim Oh, Young-Jun Lee, Jaewoo Ahn, Gunhee Kim, Dongyeop Kang

arXiv:2610.00531v1cs.AI

TL;DR

AI-generated scientific papers can look plausible while their reasoning connections break down. The paper measures these failures and grounds revisions in scientific records, reporting stronger detection than Binoculars and a smaller AI–human gap with SciSlopHarness.

  • Problem

    AI-generated content is increasingly common in academia, while scientific slop describes reasoning connections breaking down despite plausible individual parts.

    Source: §1 · p. 1

  • Method

    The paper defines six reasoning-breakdown measures across Structure, Argument, and Artifacts, benchmarks matched AI–human papers, and grounds iterative LLM revisions in scientific records.

    Source: §1 · p. 2

  • Results

    0.859 pairwise accuracy versus Binoculars’ 0.687; SciSlopHarness reduces the remaining AI–human gap by 63% over the strongest revision baseline.

    Source: §1 · p. 2, §5.2 · p. 7

  • Takeaways & Limitations

    Scientific slop is associated with weaknesses reflected in human reviews and can guide repairs grounded in scientific records.

    Source: §6 · pp. 9–10

    Evaluation is limited to FARS and Agents4Science 2025, mostly computer-science papers; broader generalization to new generators and disciplines remains untested.

    Source: §6 · p. 10

Abstract

from arXiv · show

AI-generated content, often called AI slop, is increasingly common everywhere, particularly in academia. Slop in AI-generated scientific papers, however, has more complex patterns that cannot be easily detected by existing token-based AI detectors. Each part of such a paper looks plausible while the scientific reasoning that connects the parts breaks down, which can mislead how readers assess the work. We benchmark these failures as scientific slop through six measures across Structure, Argument, and Artifacts. We construct SciSlopBench with 390 AI-generated papers, mostly in computer science but spanning the life, social, and natural sciences, each paired with a human-written paper matched by research problem and contribution type. Our measures identify the AI paper in each pair with 85.9% accuracy, compared with 68.7% for Binoculars. Higher scientific slop accompanies lower ICLR ratings and distinguishes rejected from accepted papers above chance in every year from 2017 to 2025. Reducing these patterns, however, is not as simple as directly optimizing the measures. We therefore propose SciSlopHarness, a harness-level framework that guides a fixed LLM to revise slop only where the experiment records support the change. While standard revisions leave residual slop and direct slop-aware prompting triggers reward hacking, SciSlopHarness reduces the remaining AI-human gap by 63% over the strongest revision baseline without requiring human reference targets. Overall, we demonstrate that AI-generated scientific papers leave fundamental traces in their global reasoning, and that responsible mitigation demands strict evidentiary grounding rather than mere prose refinement.

1 INTRODUCTION

The paper frames scientific slop as failures in reasoning connections that can evade token-level detection, then benchmarks and mitigates those failures. Its measures outperform a token-level baseline, while record-grounded revision reduces residual slop.

1 INTRODUCTION

  • AI-generated papers can look plausible in isolation while failures in their scientific reasoning connections evade token-level detectors.The paper identifies breakdowns among sections, claims, citations, evidence, and artifacts as paper-level failures.Source: §1 · pp. 1–2
  • SciSlopBench pairs 390 AI-generated papers with human-written papers matched by research problem and contribution type, across several disciplines.Most papers are in computer science; every human-written counterpart was accepted at a top-tier venue.Source: §1 · p. 2
  • Scientific slop measures align with human evaluations: AI-likeness falls from 26% to 12% as review scores rise from 2 to 9.The measures also distinguish rejected from accepted papers in every ICLR year from 2017 to 2025.Source: §1 · p. 2
  • Individual scientific-slop measures achieve pairwise accuracy up to 0.905 for distinguishing AI-generated from human-written papers, compared with 0.687 for Binoculars; full-text AI reviewers reach at most 0.685.SCISLOPHARNESS reduces the remaining AI–human gap by 63% over the strongest revision baseline.Source: §1 · p. 2
  • SciSlopHarness checks proposed revisions against the original manuscript and experiment records, without requiring human reference targets.General revision leaves slop unresolved, while direct slop-aware revision can lower scores without repairing reasoning.Source: §1 · p. 2

2 RELATED WORK

Prior AI-text detection focuses on local textual signals, while review and revision systems offer broader feedback without isolating recurring paper-level flaws. SciSlopHarness instead adds located findings before editing and record-grounded review afterward.

2 RELATED WORK

  • Text detectors use local features, token probabilities, or surface signals, whereas this paper measures relations across a whole paper.The cited examples include DetectGPT, Binoculars, NTS, Pangram, watermarking, and paraphrasing methods.Source: §2 · p. 3
  • Automated review and revision systems provide general feedback without isolating specific recurring patterns in papers.The paper uses AI Scientist and CycleReviewer as detection baselines and a reviewer-based refinement loop as a revision baseline.Source: §2 · p. 3
  • SciSlopHarness keeps the editor fixed, adding located findings before edits and record-grounded review afterward.The related-work framing connects this harness design to work that improves systems by changing information, checks, and tools around a fixed model.Source: §2 · p. 3

3 DEFINING AND MEASURING SCIENTIFIC SLOP

Scientific slop is defined as recurring failures in connections that let readers follow a paper’s reasoning. The paper specifies six measures and evaluates them using matched AI-generated and human-written papers.

3.2 SCISLOPBENCH

  • SciSlopBench pairs AI-generated and human-written papers matched by research problem and contribution type, then tests whether human papers score lower on slop.Each paper receives a score from the six patterns, and the paired comparison uses their mean slop scores.Source: Figure 2 · p. 4, §3.2 · pp. 4–5

3.1 SCIENTIFIC SLOP: DEFINITION AND MEASURES

  • Scientific slop comprises recurring failures in connections that hinder readers from following a paper’s scientific reasoning.The six observable patterns are grouped into Structure, Argument, and Artifacts.Source: §3.1 · p. 4
  • Structure measures missing cross-section references and macro redundancy, where later sections repeat rather than develop the argument.The measures count unreferenced sections or labeled objects and use cross-section n-gram overlap.Source: §3.1 · p. 4
  • Argument measures claims appearing before their supporting context and citations that lack an explained role in the paper’s argument.The citation measure checks whether prior work is built on, distinguished from, or compared with.Source: §3.1 · p. 4
  • Artifacts measures flag diagrams that obscure method steps and aggregate results reported without inspectable examples.The evidence-gap check applies to papers with main-text result tables and searches the paper and appendix for inputs, outputs, or cases.Source: §3.1 · p. 4

3.2.1 BENCHMARK CONSTRUCTION

  • The benchmark contains 143 FARS pairs and 247 Agents4Science 2025 pairs across computer and data, life and health, social, natural, and other sciences.It reports normalized slop rates and evaluates ranking with PairAcc and AUROC, also reporting true-positive rate at 5% false-positive rate.Source: §3.2.1 · p. 5

4 SCISLOPHARNESS: MITIGATING SCIENTIFIC SLOP

SciSlopHarness guides a fixed revision model with detected defects and revision guidance, then retains only changes supported by the manuscript and experiment records.

4 SCISLOPHARNESS: MITIGATING SCIENTIFIC SLOP

  • SciSlopHarness locates scientific slop, uses detected instances to guide manuscript revision, and reviews each proposed change against the original manuscript and experiment records.The remaining slop and review feedback guide the next revision round.Source: §4 · pp. 5–6
  • Table 3 groups scientific slop measures by plane and marks measures requiring a model; Table 2 reports paper discrimination, with TPR at 5% FPR.Table 3’s last row gives the Table 2 score.Source: Table 3 · p. 6, Table 2 · p. 6

4.1 LOCALIZED REVISION

  • The revision model receives measured problem locations and fixed guidance, but need not reduce every measure when a suggested connection is unnecessary.For example, it adds a cross-section reference only when another section uses the referenced object; no target score is specified.Source: §4.1 · p. 6

4.2 REVIEWING PROPOSED REVISIONS

  • For each changed paragraph, review checks the original manuscript, experiment outputs, and code, retaining supported changes and restoring unsupported ones.The checks cover claims, numbers, citations, and examples affected by the revision.Source: §4.2 · p. 6
  • Revision and review run for up to three rounds, stopping when no problems remain or review yields no further changes.The final manuscript is checked for preserved numbers and cited works and valid citations and references.Source: §4.2 · p. 6

5 EXPERIMENTS

Scientific slop distinguishes AI-generated papers and aligns with human review judgments; SciSlopHarness reduces residual slop while avoiding score-only revisions that lose information or break safeguards.

5.1 EXPERIMENTAL SETUP

  • The evaluation compares text detectors, automated reviewers, general revision, slop-aware revision, and SciSlopHarness under shared revision-model and round protocols.Text detectors read prose; other methods read paper source, with code access where needed.Source: §5.1 · p. 7, §5.2 · p. 7

5.2 DOES SCIENTIFIC SLOP MARK PROVENANCE, QUALITY, OR BOTH?

  • 0.859 for scientific slop exceeded 0.687 for Binoculars and 0.685 for the best automated reviewer in identifying AI-generated papers.Cross-section references alone reached 0.905; at 5% FPR, it flagged 65% of AI-generated papers versus 24% for Binoculars.Source: §5.2 · p. 7
  • AI-likeness declines more sharply with higher review scores, and scientific slop separates rejected from accepted papers above chance in all nine ICLR years.It consistently tracks lower ratings across overall rating, soundness, presentation, and contribution.Source: §5.2 · pp. 7–8

5.3 DOES SCISLOPHARNESS REMOVE THE SLOP THAT GENERAL REVISION LEAVES?

  • 63% smaller than Claude Code, SciSlopHarness’s summed six-measure gap to human means was the smallest, despite not using those means as revision targets.General revision leaves substantial slop on three measures, while slop-aware revision moves past human means; SciSlopHarness also improves on the other three measures.Source: Figure 5 · p. 8, §5.3 · p. 8

5.4 IS THE SLOP REMOVED, OR ONLY THE SCORE?

  • Table 4 contrasts score-only edits with evidence-preserving repairs: SciSlopHarness removes repetition while retaining citations and connects threshold evidence to its claim.Slop-aware revision removed four citations with a repeated sentence and scattered references without connecting most to claims.Source: Table 4 · p. 9, §5.4 · p. 9
  • 0.17 distance with no guard breaks for the full harness contrasts with 0.11 and 23% guard breaks without review, and 0.30 and 17% with review alone.The ablations indicate that locating problems, guiding repair, and checking edits must work together.Source: §5.4 · p. 9

6 DISCUSSION

The six scientific-slop measures distinguish AI-generated from human papers and are associated with lower ICLR review scores. SciSlopHarness brings the measures closest to human levels among evaluated revision methods, while the evaluation remains concentrated on two sources and mostly computer science.

6 DISCUSSION

  • SciSlopHarness brings all six measures closest to human levels among evaluated revision methods, without human reference targets.Its revisions are checked against the paper’s experiment outputs and code.Source: §6 · pp. 9–10
  • The authors introduce scientific slop as measurable breakdowns in reasoning connecting a paper’s parts; its measures distinguish AI-generated from human papers and are associated with lower scores across all four ICLR review dimensions.Source: §6 · p. 9

Limitations and future directions

  • The evaluation focuses on FARS and Agents4Science 2025, with most papers drawn from computer science.The authors suggest extending paired evaluation to new generators and disciplines to test how broadly the findings apply.Source: §6 · p. 10

AI USE STATEMENT

The authors disclose using LLMs as both research subjects and experimental components, as well as for implementation and manuscript preparation. They report reviewing and testing assisted code and reviewing the final manuscript.

AI USE STATEMENT

  • The study uses third-party AI-generated papers, Qwen2.5 models for measurement, and Claude models as editors and reviewers.Claude, Claude Code, and ChatGPT also assisted with code and manuscript preparation.Source: AI USE STATEMENT · p. 10
  • The authors used AI tools to write and debug code, edit the manuscript, and in some cases produce initial text that they later edited.The tools assisted with the measurement pipeline, harness, baselines, and analysis scripts.Source: AI USE STATEMENT · p. 10
  • The authors reviewed and tested assisted code, reviewed the final manuscript, and obtained all reported results by running the released code.Source: AI USE STATEMENT · p. 10

ETHICS STATEMENT

The ethics statement frames the work as supporting scrutiny of reasoning failures without making claims about author intent or misconduct. It also describes record-grounded revision checks and reports aggregate relationships between measured slop and review outcomes.

ETHICS STATEMENT

  • Scientific-slop measures give authors and reviewers concrete points to examine, including weaknesses that fluent prose can obscure.The findings concern paper reasoning and do not establish author intent or misconduct.Source: ETHICS STATEMENT · p. 10
  • SciSlopHarness keeps a revision only when the manuscript and experiment records support it, and records citation and number preservation checks.When records contain no concrete example, the revision acknowledges that absence.Source: ETHICS STATEMENT · p. 10
  • The ICLR analysis reports aggregate relationships between measured slop and review outcomes using public manuscripts and review information.The benchmark’s data sources and processing procedures are documented.Source: ETHICS STATEMENT · p. 10

A MEASUREMENT SPECIFICATION OF THE SLOP MEASURES

The appendix specifies paper-level measures and how SciSlopBench pairs AI-generated papers with accepted human-written anchors. It details the pairing procedures for FARS and Agents4Science, including document reconstruction for submissions lacking source files.

A MEASUREMENT SPECIFICATION OF THE SLOP MEASURES

  • The structure view preserves sections, labels, references, citations, and floats, while the prose view removes floats and captions and replaces mathematics and references with sentinels.Macro redundancy marks a sentence as recycled when more than half its tokens lie in 8-grams already used in an earlier section.Source: A MEASUREMENT SPECIFICATION OF THE… · p. 14
  • The argument graph marks key claims when their supporting context comes after them, while citation isolation distinguishes weaving citations from isolated ones.Key claims include sentences labeled superiority, prior limitation, or design choice.Source: A MEASUREMENT SPECIFICATION OF THE… · p. 15
  • Figure exposition checks six kinds of content in a method-figure transcript, while evidence gap applies to papers with result tables meeting specified size criteria.An exhibit anywhere in the paper, including the appendix, closes the evidence gap.Source: A MEASUREMENT SPECIFICATION OF THE… · p. 15
  • The SciSlop score averages applicable item scores within each plane and then averages the planes.The reported SciSlop row uses the negative of the score wherever higher values should favor human authorship.Source: A MEASUREMENT SPECIFICATION OF THE… · p. 15

B SCISLOPBENCH CONSTRUCTION DETAILS

  • SciSlopBench contains 390 AI-generated and human-written paper pairs, matched by contribution type and the same or a neighboring research problem.Each human-written anchor has verifiable human authorship and peer acceptance.Source: B SCISLOPBENCH CONSTRUCTION DETAILS · p. 16

B.1 PAIRING RULE

  • Human anchors must meet authorship, acceptance, contribution-type, and usable-source conditions, with candidates selected by problem similarity and recorded tie-breakers.The matching procedure searches cited papers first, then a widened pool of accepted ICLR papers when needed.Source: §B.1 · p. 16

B.2 FARS: COLLECTION AND MATCHING

  • The FARS portion contains 143 pairs: 120 from cited papers and 23 from the widened pool.Some FARS papers had no same-type anchor in either pool and were not paired.Source: §B.2 · p. 16

B.3 AGENTS4SCIENCE: COLLECTION, MATCHING, AND DOCUMENT PROCESSING

  • All 247 Agents4Science submissions receive human anchors, though only 185 meet the full pairing rule; 229 submissions are reconstructed from PDF.Overall, 206 anchors match contribution type and 196 match the research problem; every pair’s match level is recorded.Source: §B.3 · p. 17

B.4 COMPARISON WITH EXISTING BENCHMARKS

SciSlopBench preserves complete papers and matched scientific context to evaluate failures that passage-level benchmarks cannot represent. Its design supports measuring relations across sections, claims, citations, figures, tables, and evidence.

B.4 COMPARISON WITH EXISTING BENCHMARKS

  • SciSlopBench pairs complete AI-generated papers with human papers on the same research problem, making cross-paper scientific context and relationships measurable.The FARS portion also includes code used to produce reported results, enabling checks between manuscript claims and experimental records.Source: §B.4 · p. 17
  • Tables 6 and 7 classify the 247 Agents4Science 2025 submissions by primary and secondary topics and accepted or rejected status.Table 7 notes that rejected counts are calculated as total minus accepted counts.Source: Table 6 · p. 18, Table 7 · p. 18
  • Table 8 compares benchmark properties, marking whether each property is present, partial, studied but absent from instances, or absent.Partial coverage includes cases such as a human text from the same source rather than a matched counterpart.Source: Table 8 · p. 18

C SCISLOPHARNESS IMPLEMENTATION DETAILS

The appendix specifies SciSlopHarness’s revision, review, measurement, and stopping procedures, then describes four baselines run under a shared protocol. The reported comparison finds the harness closest to human means across all six measures.

C SCISLOPHARNESS IMPLEMENTATION DETAILS; C.1 CONFIGURATION

C.2 LOCALIZED REVISION

  • Localized revision gives the editor pattern definitions, repair directions, and located instances, while prohibiting fabricated evidence and providing no score or target.The editor works on one source file per call; the controller also copies experiment records and checks for a displayable specimen quoted from a named file.Source: §C.2 · p. 19

C.3 REVIEW AGAINST SCIENTIFIC RECORDS

  • Reviewers assess each before-and-after change against the manuscript and project records, reverting unsupported claims or altered numbers and keeping only explicitly approved edits.A change that mixes supported and unsupported edits is reverted in full; retirement removes instances from feedback but not from measurement.Source: §C.3 · p. 20

C.4 MEASUREMENT, PRESERVATION, AND STOPPING •

  • After review, retained manuscripts are remeasured; preservation checks can disqualify a round, and revision stops when no instances remain, edits cease, two rounds are fully reverted, or three rounds finish.Reverted edits do not contribute to the reported reduction.Source: §C.4 · p. 21, Table 11 · p. 21

C.5 DISTANCE TO HUMAN MEANS

  • 63%: SciSlopHarness closes the six-measure gap to human means from 1.395 to 0.519, ending closest to the human mean on every measure.The comparison uses Claude Code, the smallest-gap baseline; slop-aware revision passes below human means on three measures, causing its summed gap to grow over rounds.Source: Table 12 · p. 21, §C.5 · p. 21

D REVISION BASELINE DETAILS; D.1 SHARED PROTOCOL; D.2 BASE PROMPTING; D.3 CLAUDE CODE

  • All baselines share the editor model, manuscript state, three-round protocol, and measurement, and receive neither human references nor slop scores or formulas.Base prompting edits the complete LaTeX source in a prompt; Claude Code edits files in restricted mode without shell or code execution.Source: D REVISION BASELINE DETAILS · p. 22, §D.1 · p. 22, §D.2 · p. 22

D.4 REVIEWER-BASED REFINEMENT

  • Reviewer-based refinement regenerates a structured review from each current manuscript before an editing session, using general prompts that do not mention SciSlop or measured quantities.The manuscript supplied to the review model is truncated to 50,000 characters, with shorter retries if valid JSON is not returned.Source: §D.4 · pp. 23–24

D.5 SLOP-AWARE REVISION

  • Slop-aware revision supplies detected patterns, repair instructions, and manuscript locations instead of a general review, without giving the editor score formulas or target scores.It lists up to 12 instances per pattern and skips a round when no problems are detected.Source: §D.5 · p. 24, Table 13 · p. 25

E ICLR DATA AND MEASUREMENT

The appendix describes ICLR corpora and measurement procedures used to relate AI-likeness scores to ratings, review dimensions, and decisions. Measures and calibration were fixed on SciSlopBench, with statistics computed within year because the rating scale changed.

E ICLR DATA AND MEASUREMENT

  • The measurers and calibration were fixed on SciSlopBench, and statistics are computed within each year or within ICLR 2026 because rating scales changed over time.Figure 4 uses 1,177 multi-year papers and 691 ICLR 2026 papers scored with common scripts, settings, and document views.Source: E ICLR DATA AND MEASUREMENT · p. 26
  • The multi-year corpus contains 1,177 ICLR 2017–2025 papers after 204 of 1,381 decided papers failed extraction rules; the ICLR 2026 corpus contains 691 submissions with arXiv e-prints.The multi-year set includes 590 rejected, 434 accepted, and 153 oral papers; the 2026 set includes 560 accepted and 131 rejected.Source: E ICLR DATA AND MEASUREMENT · p. 26
  • Exclusions depend only on source measurability criteria, such as missing inputs, failed parsing, too few sections or objects, short prose, or insufficient measure denominators.ICLR 2026 also excludes papers with fewer than three reviews or body length outside the 1st–99th percentile; removed papers are excluded for every system.Source: E ICLR DATA AND MEASUREMENT · p. 26
  • Detectors score prose, reviewers read body source, and measures use full source; scores are oriented so higher values indicate more AI-like writing.Argument graph was not run on ICLR 2026, while Evidence gap and Figure exposition apply only to subsets of the multi-year corpus.Source: E ICLR DATA AND MEASUREMENT · p. 26
  • Panel (a) applies calibration learned on SciSlopBench unchanged to ICLR papers and plots AI-authorship probabilities by mean rating, adjusted for year.Panel (b) shows within-paper-set Spearman correlations with ICLR 2026 reviewer dimensions; panel (c) compares the within-year probability that rejected papers score more AI-like than accepted or oral papers.Source: E ICLR DATA AND MEASUREMENT · p. 26

F ADDITIONAL MATCHED REVISION CASES

Across four additional matched revision cases, SciSlopHarness repairs located passages while slop-aware revision lowers scores by leaving, deleting, or inventing content. The cases illustrate distinct repairs and an image-editing tool asymmetry.

F ADDITIONAL MATCHED REVISION CASES

  • For Citation isolation, SciSlopHarness adds a contrast between CALM and InfMem, whose chunk-level method reports a 3.3–5.1× speedup; rewording CALM alone leaves it isolated.The score reduction comes from adding a relation, whereas slop-aware revision changes elsewhere in the paper.Source: F ADDITIONAL MATCHED REVISION CASES · p. 27
  • For Evidence gap, SciSlopHarness acknowledges aggregate-only claims and reverts an unsupported worked example, leaving the gap acknowledged at 0.5 rather than filling it with invented details.Slop-aware revision reaches zero by inserting an example absent from both manuscript and project records.Source: F ADDITIONAL MATCHED REVISION CASES · p. 27
  • For Argument graph, SciSlopHarness moves the design claim directly after its method description, reducing the manuscript score to zero by reordering existing content.Slop-aware revision leaves the claim ahead of the method and adds a section pointer instead.Source: F ADDITIONAL MATCHED REVISION CASES · p. 27
  • For Figure exposition, SciSlopHarness erases six of eight redundant phrases while preserving diagram components and arrows, but the comparison is unequal because the baseline lacks an image editor.The source-text-only baseline leaves all eight phrases in the diagram and its score unchanged.Source: F ADDITIONAL MATCHED REVISION CASES · p. 27
  • Table 14 and the four cases show slop-aware revision lowering scores by leaving, deleting, or inventing content, while SciSlopHarness repairs the located passage itself.The comparison extends the earlier two-measure analysis to Citation isolation, Evidence gap, Argument graph, and Figure exposition.Source: F ADDITIONAL MATCHED REVISION CASES · p. 27, Table 14 · p. 28
Loading 2610.00531v1…