PaperlayerView exact revision on arXiv

Source-linked AI summary

Multimodal AI Detection for Math-Heavy Scientific Text

Yixuan Huang, Danish Pruthi, Bhuwan Dhingra

arXiv:2610.21062v1cs.CL

TL;DR

Math-heavy scientific writing poses a detection challenge because PDF text extraction loses mathematical formatting, even as accurate AI-use disclosure matters for research. The paper builds paired human and LLM-generated proof and prose data across three representations and trains detectors including SnapJudge. Its detectors achieve over 90% detection at a 0.1% false-positive rate, and SnapJudge outperforms Pangram 4 on frontier-model proofs.

  • Problem

    PDF text extraction damages mathematical formatting, while accurate detection matters for verifying AI-use disclosure and assigning research credit.

    Source: §1 · pp. 1–2

  • Method

    The authors pair human-written proofs and prose with LLM regenerations and train detectors for LaTeX, PDF-extracted text, and rendered images.

    Source: §1 · p. 2

  • Results

    SnapJudge outperforms Pangram 4 by 26.2 percentage points on frontier-model proofs, while the detectors exceed 90% detection at a 0.1% false-positive rate.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    Image-based detection approaches LaTeX performance and outperforms the text-based detector, while also performing well on several datasets beyond mathematics.

    Source: §1 · p. 2

    The detectors cannot yet reliably distinguish AI-polished text from fully AI-generated text.

    Source: §6 · p. 10

Abstract

from arXiv · show

As large language models become increasingly capable of generating, and assisting with, scientific writing, there is a need for greater transparency and disclosure around the use of AI in research. Consequently, a veritable subfield of research and a number of commercial services aim to flag AI-generated content automatically. However, existing detectors which rely only on text extracted from PDFs fail to benefit from the rich visual information contained in the formatting of math-heavy scientific content. To study this, we select human-written proof and prose passages from pre-2022 arXiv LaTeX sources and re-generate them using 15 modern LLMs within the context of the same paper. This leaves us with coupled human and AI outputs in three different representations: the LaTeX source, a rendered image of the PDF, and the text extracted from the PDF. We find that the accuracy of the state-of-the-art Pangram 4 drops significantly as the mathematical content in a passage increases, especially when evaluated on text as opposed to LaTeX. However, this is not an inherent limitation of detecting AI-generated math: finetuning Qwen-3.5-27B based detectors on this domain recovers the gap, with LaTeX inputs performing significantly better than text since they preserve more formatting information. Since LaTeX sources are rarely available in practice, we also train SnapJudge: an image-based detector operating on screenshots of both prose and proofs from papers. At 0.1% false positive rate, SnapJudge achieves a detection rate of 92% on proofs and 93% on prose generated by frontier models, substantially outperforming both Pangram 4 and our text-based detector. We release our data, models and code at https://nlplab.cs.duke.edu/snapjudge.

1. Introduction

The introduction argues that accurate detection is needed to verify AI-use disclosures, but math-heavy scientific writing challenges detectors that rely on extracted text. It presents paired human and LLM-generated passages and compares detection across LaTeX, extracted text, and rendered images.

1. Introduction

  • Accurate detection matters because publication policies require AI-use disclosure, and verification supports assigning credit even when a paper's claims are correct.The introduction says honest disclosure can only be verified and enforced when accurate detection is available.Source: §1 · p. 1
  • The introduction frames the central question as whether detectable signs persist in math-heavy writing, where formal constraints limit how ideas can be expressed.It reports that AI-generated math remains detectable under all three representations.Source: §1 · p. 1
  • Readers seldom have the LaTeX source; extracting PDF text damages mathematical formatting, while OCR can add formatting signals that impede detection.The introduction identifies these as the two common ways prior detectors handle PDFs.Source: §1 · pp. 1–2
  • At a 0.1% false-positive rate, the detectors exceed 90% detection; SnapJudge approaches LaTeX accuracy and outperforms the text-based detector.The image-based detector also performs well beyond mathematics, including on MIRAGE, RAID, AI-written peer reviews, and EditLens.Source: §1 · p. 2
  • At a 0.01% false-positive rate, SnapJudge detects more than 82% of AI-written passages, including over 80% of proofs composed primarily of formulas and equations.Passages become harder to detect as their formula share increases.Source: §1 · p. 2
  • The study pairs 47K human-AI proofs and 45K human-AI prose passages, each represented as LaTeX, PDF-extracted text, and a rendered image.The passages are drawn from pre-2022 arXiv papers and regenerated with modern LLMs conditioned on the rest of each paper.Source: §1 · p. 2
  • LaTeX detectors perform best, text-based detectors struggle with math-heavy text, and image-based detectors approach LaTeX performance; SnapJudge beats Pangram 4 by 26.2 percentage points on frontier-model proofs.The introduction also reports rising AI use in mathematics, reaching 23.1% of prose and 17.6% of proofs in August 2026.Source: §1 · p. 3

2. Related Work

Related work describes text detectors developed chiefly for natural-language prose, image-based document processing, and the distinct task of detecting AI-synthesized images. The paper positions math-heavy scientific writing as a setting where formatting matters and evaluates detection of AI-written text shown in document images.

2. Related Work

  • Existing text detectors score language-model statistics or classify labeled examples and are primarily developed and evaluated on natural-language prose; off-the-shelf performance is limited, especially on proofs.Pangram 4 reports above 99% detection at a 0.004% false-positive rate on its own evaluation.Source: §2 · p. 3
  • Document-image research includes pixel-based language modeling, image retrieval, and rendering text into images; prior scientific retrieval results favor LaTeX over whole-page images.The cited retrieval work reports that image performance degrades more sharply as documents grow longer.Source: §2 · p. 3
  • Most image-based detectors ask whether an image was AI-synthesized, not whether its displayed text was AI-written; AIDoc evaluates earlier open-weight generators but not low-false-positive deployment.OCR-recovered text also embeds signatures that are difficult to detect in math-heavy content.Source: §2 · p. 3

3. Setup & Methods

The study builds paired human–AI proof and prose data from pre-2022 arXiv papers and evaluates detectors across LaTeX, extracted text, and rendered images. It also tests real-page crops, multiple degrees of AI editing, and baselines under paper-level splits and calibrated thresholds.

3.1. Dataset

  • Human references come from 2019–2021 arXiv papers never revised afterward, with proofs defined as theorem-proof bodies and prose as paragraphs without theorem, proof, figure, or table content.Prose may contain inline mathematics and other LaTeX content.Source: §3.1 · pp. 3–4
  • The dataset pairs human proofs and prose with regenerations from 15 LLMs, represented as LaTeX, PDF-extracted text, and rendered images.Each target passage is regenerated in the style of its source paper using the rest of that paper as context; compiling standalone passages succeeds for 94% of pairs.Source: §3.1 · p. 4
  • Real-page crops place generated passages among human-written context, with an AI-text fraction used to train a fraction estimator and pure-AI or pure-human crops added to detector training.These crops are intended to improve robustness to page artifacts such as line numbers and watermarks.Source: §3.1 · p. 4
  • The test set also includes sentence-level edits, whole-passage rewrites, and regenerations from logical-step outlines, while data splits are by paper and thresholds use human calibration pools.Train and validation data are mainly from 2019–2020 papers; test data are from 2021 papers.Source: §3.1 · p. 4
  • The study acknowledges contamination risk from LLM pretraining, removing 0.3% of generations with over 10% word-level 8-gram overlap while noting that detector pretraining contamination remains a shared supervised-detector concern.The authors note that online text produced since 2022 cannot be guaranteed entirely human-written, so most detectors rely on pre-2022 content.Source: §3.1 · pp. 4–5

3.2. Detectors

  • Separate passage-level binary detectors process LaTeX, extracted text, or images, alongside an image-based estimator of the AI-generated fraction.A passage is flagged when its predicted probability of LLM generation exceeds a threshold.Source: §3.2 · p. 5
  • The detector models use Qwen3.5 with LoRA fine-tuning; text inputs use overlapping 512-token windows, whereas SnapJudge takes rendered images as input.Image encoding uses 32×32-pixel patches with a budget of about 1.25 megapixels.Source: §3.2 · p. 5

3.3. Baselines

  • Comparisons include logistic-regression baselines using stylometric, character n-gram, and word n-gram features, plus Binoculars, Fast-DetectGPT, EditLens, and Pangram 4.The shallow baselines are trained separately by representation and by proof versus prose.Source: §3.3 · p. 5

4. Results

Across the three representations, image-based SnapJudge approaches LaTeX detectors and outperforms text-based detection on frontier passages; detection remains robust across mathematical density, unseen generators, and external domains. The analysis also examines AI-edited passages and estimates AI-writing trends in arXiv papers.

4.1. Detection rates under the three representations

  • At 0.1% FPR, SnapJudge’s frontier-family averages are 0.918 on text and 0.928 on images, versus Pangram 4’s 0.656 and 0.871, respectively.Table 2 compares LaTeX-source detection with methods using PDF-derived inputs; SnapJudge is the most accurate where LaTeX is unavailable, especially on frontier proofs.Source: Table 2 · p. 6, §4.1 · p. 6
  • At 0.001% FPR, SnapJudge still detects above 0.67 of proofs and 0.80 of prose; detection remains above 75% across math-fraction bins.Removing or replacing math in the LaTeX input sharply lowers detection, and detectors still identify most correct AI proofs and equivalent AI passages; incorrect human proofs are not flagged at 0.1% FPR.Source: §4.1 · p. 7, §4.2 · pp. 7–8

4.3. Generalization to unseen generators and domains

  • On held-out frontier models, SnapJudge loses 6.8 points on proofs and 4.1 on prose, versus 11.0 and 6.9 points for the LaTeX detector.Across four external benchmarks, false-positive rates are no higher than 0.02%, with detection rates generally on par with the best methods from the respective papers; SnapJudge also has the lowest expected calibration error on those benchmarks and the paper’s test sets.Source: §4.3 · p. 8, Table 4 · p. 8

4.4. Detection of AI-edited passages

  • SnapJudge’s scores correlate with AI-editing degree about as strongly as EditLens, which is trained to predict that degree, and more strongly than zero-shot detectors.Because mixed datasets have no fixed positive-negative boundary, users can set a threshold according to how much AI editing they accept.Source: §4.4 · p. 9

4.5. AI-generated text on arXiv over time

  • By August 2026, 17.6% of arXiv proofs were detected as AI-generated, with use in mathematics growing rapidly since its later arrival.For 2018, 2019, and 2021, false-positive rates were 0.008% for proofs and 0.003% for prose, below the detector’s 0.01% calibration threshold.Source: §4.5 · p. 9

5. Conclusion

The paper presents a paired dataset of human-written and AI-generated proofs and prose, and reports that its detectors can distinguish them across representations. It also reports generalization to unseen models and domains, rising AI use in mathematical arXiv content, and directions for future detector development.

5. Conclusion

  • Figure 4 tracks the share of arXiv passages flagged by the LaTeX detector at a 0.01% false-positive rate.The passage describes the figure but does not provide its plotted values.Source: Figure 4 · p. 9
  • The dataset pairs human-written and AI-generated proofs and prose across LaTeX, PDF-extracted text, and rendered images.The paper identifies this dataset as the basis for its experiments across three input representations.Source: §5 · p. 9
  • The experiments consistently find AI-generated math-heavy text detectable, and the detectors outperform Pangram 4 on scientific text.The paper also reports generalization to unseen models and domains, with detector scores correlated with the extent of AI editing.Source: §5 · p. 9
  • The authors report that AI use in mathematical content on arXiv is rapidly rising.They identify multi-representation detection and finer-grained quantification of AI contributions as directions for future work.Source: §5 · p. 9

6. Limitations

The detectors do not yet reliably distinguish AI-polished text from fully AI-generated text. SnapJudge is trained on proof and prose passages, not figures or tables.

6. Limitations

  • At a 0.005% false-positive rate, the detectors flag about 17% to 42% of human-written but AI-polished peer reviews as AI.Tightening the threshold further leads to a large drop in detection rates for purely AI-generated categories.Source: §6 · p. 10
  • SnapJudge is trained on proof and prose text passages and cannot yet judge more multimodal content such as figures and tables.Combining judgments of figures and tables with text is left for future work.Source: §6 · p. 10

7. AI Use Statement

The authors describe AI assistance across coding, experiments, related-work organization, writing, figures, tables, and prompts, while retaining responsibility for the final work.

7. AI Use Statement

  • The authors proposed the research questions and experiments; some ideas emerged from discussions with AI.Source: §7 · p. 10
  • Claude Opus 5, Fable 5, and Opus 5.5 helped write code, schedule training jobs, and compute experimental results, which the authors manually reviewed and checked.Source: §7 · p. 10
  • Opus 5 helped search for and organize related work; it and Opus 5.5 polished most of the authors' first-draft main text, which the authors further revised.Source: §7 · p. 10
  • Some passages were drafted in Chinese, translated into English by Opus 5 and Opus 5.5, and then revised by the authors.Source: §7 · p. 10
  • Opus 5 and Opus 5.5 generated the figures, tables, and first appendix draft for author review and revision; Opus 5 and Fable 5 generated prompts from author-provided key points.Source: §7 · p. 10
  • The authors state that they take full responsibility for the final content, including text, claims, and AI-assisted artifacts.Source: §7 · p. 10

A. Training Details and Hyperparameters

The appendix provides training-setting and detector-performance materials, including a comparison of detection rates on short passages.

A. Training Details and Hyperparameters

  • Table 6 lists training settings for the 27B models behind the main results and the 9B detectors used in Section 4.3.Source: Table 6 · p. 15

B. Detection Rates by Generator Family

C. Detection Rates by Passage Length

  • Below 100 words, the detectors still detect more than 80% of proofs and 88% of prose, while Pangram 4 depends much more on passage length.The comparison uses frontier test passages and thresholds from Table 2; Figure 5 plots detection rate at 0.1% FPR against AI-generated passage length for Pangram 4 on text.Source: C. Detection Rates by Passage Length · p. 17, Figure 5 · p. 17

D. Detection Rates on AI-Edited Datasets

Table 9 reports detection rates across datasets and editing conditions, with different false-positive-rate settings for off-the-shelf and paper-trained detectors.

D. Detection Rates on AI-Edited Datasets

E. Detection of AI-Polished Peer Reviews

On peer-review data, the detectors generally identify LLM-written reviews but struggle to distinguish polished human reviews from AI-written ones. Tests on newer models find improved detection after limited retraining, while conventional AI-image detectors perform at chance on rendered pages.

E. Detection of AI-Polished Peer Reviews

  • At 0.005% FPR, detectors had no false positives on human-written reviews and generally detected nearly all AI-written reviews, but often labeled LLM-polished human reviews as AI.The benchmark labels polished reviews as Human in line with policies that permit LLM use for polishing; the authors describe the labeling as subjective and dependent on the degree of polishing.Source: E. Detection of AI-Polished Peer Reviews · p. 18, Table 10 · p. 19
  • Eliminating flags on polished human reviews also caused a catastrophic drop in detection of AI-written reviews, indicating difficulty separating the categories.The threshold was raised until the share of 2,802 polished reviews flagged reached the specified level.Source: E. Detection of AI-Polished Peer Reviews · p. 19, Table 11 · p. 19

F. Detection of Newer Models

  • For GPT-6 and Claude Opus 5.5, detection fell relative to other models, but adding data from those models—about 1.5% of training data—clearly improved it, with no human test passages flagged.Pangram 4 performed poorly on both newer models and labeled at most a quarter of passages as AI or mixed.Source: F. Detection of Newer Models · p. 19, Table 12 · p. 20

G. AI-Generated Image Detectors on Rendered Pages

  • Five open-source detectors designed to identify AI-generated images scored at chance on the rendered pages, which were real compiler output differing only in text authorship.The detectors were evaluated on pages from both test batteries, averaging scores over document crops.Source: G. AI-Generated Image Detectors on… · p. 20, Table 13 · p. 20

H. Monthly Data of the arXiv Trend

The arXiv trend analysis samples proof and prose passages across years and fields, while a resolution test examines SnapJudge on lower-quality screenshots. Detection on proofs declines as resolution falls, with sharper drops below 40 dpi.

H. Monthly Data of the arXiv Trend

  • The arXiv trend sample tracks monthly shares of prose and proofs flagged by the LaTeX detector at 0.01% FPR, with yearly summaries by field.Sampling spans 2018–2026, skips 2020, and expands to computer science and other fields from 2023 onward; the supplied passages describe the tables but do not provide their values.Source: H. Monthly Data of the arXiv Trend · p. 20, Table 14 · p. 21, Table 15 · p. 22

I. SnapJudge at Lower Resolution

  • Proof detection fell from 91.8% at 150 dpi to 81.8% at 50 dpi, while prose detection barely changed down to 40 dpi.The authors suggest proofs are more sensitive because subscripts and small formula symbols become illegible first.Source: §I · p. 22, Figure 6 · p. 22
  • Below 40 dpi, detection dropped sharply for both prose and proofs as the text itself became difficult to read.The test used frontier passages rendered from 150 dpi down to 20 dpi, with the threshold unchanged.Source: §I · p. 22, Figure 6 · p. 22

J. Proof-Carrying Submissions at ICLR

Proof-carrying ICLR submissions are associated with higher review scores and acceptance rates. The appendices also document how passages and comparison prompts were generated for the study’s dataset and equivalence judging.

J. Proof-Carrying Submissions at ICLR

  • Proof-carrying submissions had higher review scores in all seven ICLR years from 2020–2026, with significant differences, and higher acceptance rates in every year, significant in six.The acceptance-rate difference in 2021 was not significant (z = 1.8); for the other reported comparisons, the table caption defines significance as |z| > 1.96.Source: J. Proof-Carrying Submissions at ICLR · p. 22, Table 16 · p. 23
  • The share of proof-carrying papers is rising, especially in theory-oriented areas, where the acceptance premium was 11 to 15 percentage points each year.The reported premium covers nine areas and has p < 2 × 10^-13.Source: J. Proof-Carrying Submissions at ICLR · p. 23, Table 17 · p. 23

K. Generator Models and Generation Prompts

  • Generation prompts place model-written passages into their paper context and require the output to match the paper’s style, notation, and LaTeX conventions.The frontier-model passages cost about $0.03 each, or about $390 for the frontier half of the dataset; open-source generators ran on the authors’ GPUs without API cost.Source: K. Generator Models and Generation… · pp. 23–25

L. Prompts of the Equivalence Judge

  • The equivalence judge compares paragraph content while ignoring presentation, and evaluates proof validity separately from similarity of mathematical strategy.The prompts use a 1–10 similarity scale and instruct judges not to infer authorship from style, polish, or length.Source: L. Prompts of the Equivalence Judge · pp. 25–27

M. Validation of the Proof Judge

The paper evaluates a proof judge, compares detector calibration, and assesses an estimator of the AI-generated fraction in image windows. The proof judge’s severity score distinguishes flawed proofs from controls, while SnapJudge is reported as less miscalibrated than a text-based detector.

M. Validation of the Proof Judge

  • A flawed proof received a higher severity score than a control proof with probability 0.81 (one-sided Mann-Whitney test, p=0.0016).GPT-5.6 Sol scored proofs from withdrawn ICLR 2026 submissions flagged by reviewers and reviewer-described correct control proofs.Source: Table 19 · p. 27
  • The judge re-derives and enumerates proof issues, ranks them, then scores only the worst issue according to the repair required.The repair categories distinguish cosmetic issues, one-line repairs, nontrivial arguments, and counterexamples for false claims.Source: M. Validation of the Proof Judge · p. 28
  • The judge is instructed to treat plausibly established elsewhere hypotheses as supplied and to support each reported defect with a verbatim proof excerpt.A defect’s severity depends on the repair needed, not the number of findings.Source: M. Validation of the Proof Judge · pp. 28–29

N. Calibration of the Detector Scores

  • SnapJudge is less miscalibrated than the text-based detector on in-domain and external tests and alleviates overconfidence on unfamiliar human-written text.Reliability diagrams bin detector scores into eight ranges; plotted points compare mean score with the share of AI samples, with AI samples weighted to match human-written samples within each dataset.Source: N. Calibration of the Detector Scores · pp. 29–30, Figure 7 · p. 30

O. AI-Fraction Estimator

  • The AI-fraction estimator predicts the AI-generated share of image text by height using five buckets and achieves 0.885 bucket accuracy on the test set.The reported fraction estimate is a probability-weighted average of bucket midpoints, and the estimator is applied only to passages SnapJudge has already judged AI-generated.Source: O. AI-Fraction Estimator · p. 31
  • On 5,000 framed test windows, mean absolute fraction-estimation error was 0.055 for the estimator alone and 0.051 for the cascade.The cascade first gates windows with SnapJudge at a 0.1% false-positive threshold, then assigns a bucket to passed windows.Source: Table 21 · p. 31, O. AI-Fraction Estimator · p. 31
  • The estimator’s input windows can contain both an AI-generated proof and surrounding human-written context, with AI fraction measured by text height.Figure 8 illustrates sampled windows that fully cover the AI passage together with part of the surrounding human-written text.Source: Figure 8 · p. 32

Related papers

Loading 2610.21062v1…