How Much Were You Told? Measuring External Information in Peer Reviews

Policy context

• Current policies permit some language assistance but prohibit delegating criticism, leaving a difficult boundary between machine-made wording and machine-supplied review content.ICLR, NeurIPS, and *ACL generally permit grammar or clarity assistance; ICML 2026 permits privacy-compliant comprehension help but not generated criticisms.Source: §1 · p. 1

The detection gap

• Artificial Text Detection can flag both human critiques polished by an LLM and critiques generated by an LLM, although only the latter delegates substantive reviewing work.ATD measures whether a model shaped surface text, while policy compliance turns on who supplied the content.Source: §1 · p. 2

Contribution

• The paper defines external information as review content unexplained by the paper and generic instructions, estimated with an information-theoretic proxy-model criterion.The measure is source-agnostic: additional information may come from a human, retrieval, or another system.Source: §1 · p. 2
• Self-Conditioning is proposed as an unsupervised estimator that separates fully delegated from machine-polished reviews and varies monotonically with controlled external information.The authors also characterize important edge cases and release code and data.Source: §1 · p. 2

Peer reviewing, NLP, and AI

• PeerRead and NLPeer support computational peer-review studies, alongside research on unconstructive reviewing, review-derived inquiries, and LLM-generated reviews.Related work also reports that LLMs can struggle with methodological novelty and replicate human biases.Source: §2 · p. 2

AI-generated text detection

• Artificial-text detection spans human-versus-machine, hybrid-text, generator-specific, and source-attribution tasks, with supervised and zero-shot approaches.Supervised detectors may degrade under shifts in generator, prompt, domain, or decoding.Source: §2 · p. 2

Editing and adversarial evaluation

• Hybrid-text methods examine rewriting discrepancies or AI editing, while benchmarks expose sensitivity to adversarial prompting and paraphrasing.Xie et al. study mutual information between generated text and its known prompt.Source: §2 · p. 2

Positioning

• Self-Conditioning estimates externally supplied information relative to production context, unlike provenance, source-attribution, or AI-editing measures.Its framing draws on coding interpretations of language-model likelihoods and information-theoretic prompt selection.Source: §2 · p. 2

3.1 Review-production regimes

• The evaluation defines human, machine-polished, partially informed, and fully delegated review regimes, and uses controlled settings because no standard benchmark covers the formulation.Partially informed reviews use k human-review bullets; fully delegated reviews use the paper, conference guidelines, and final decision.Source: §3.1 · pp. 2–3

3.2 External information as context-relative information gain

• The baseline context C contains the paper and generic reviewing instructions, while latent U represents information supplied beyond C regardless of its provenance.U may reflect human reasoning, retrieval, tools, or other sources; the aim is to quantify review information unexplained by C, not identify its author.Source: §3.2 · p. 3
• Because U is unobserved, the method reveals a fixed-budget fragment X sampled from the review and measures its explanatory value for that review.The ideal score compares the review’s likelihood with and without X, normalizes by review length, and treats the gain as reduced description length.Source: §3.2 · p. 3
• Self-Conditioning information is framed as I(T; X | C), with residual variation capturing unpredictability that remains even given C and U.Substantive critiques not recoverable from C can raise the score, but stochastic generation can too; the score is therefore a graded proxy, not a provenance or cognitive-effort detector.Source: §3.2 · p. 3

3.3 Practical score

• The practical score estimates per-token likelihood gain from review-derived hints under a proxy language model, averaging over sampled hints.For peer review, the context is the paper plus generic reviewing instructions, and the assessed text is the review.Source: §3.3 · pp. 3–4
• The practical comparison is relative: hints are hypothesized to provide more likelihood gain when review content is not predictable from the baseline context.The score uses Random Words by default, revealing 30% of review words in their original order and averaging 10 draws.Source: §3.3 · p. 4

4.1 Data and models

• IntelLabs-500 contains 500 pre-2022 ICLR 2019 human reviews and five generated counterparts, with additional fully delegated reviews generated by GPT-5.4 and Gemini-3.1-Pro.Machine-polished reviews use Llama-3.1-8B-Instruct or Qwen3-8B; partially informed reviews use k of eight structured human-review bullets.Source: §4.1 · p. 4

4.2 Evaluation protocol

• The evaluation tests separation of delegated reviews from human and polished reviews, score changes across the partially informed ladder, and robustness to uninformative humans and surprising machine texts.Pairwise AUC measures separability; the protocol expects AUC against human reviews to decrease as more human-derived bullets are supplied.Source: §4.2 · p. 4

5.1 Polished reviews score like human ones (RQ1)

• Machine-polished reviews remained close to human reviews under Self-Conditioning, with AUC between 0.556 and 0.610, even when the polishing model was small.Fully delegated reviews from older or smaller models were easier to distinguish from human writing; Gemini-3.1-Pro and GPT-5.4 overlapped more with the human distribution.Source: §5.1 · p. 5, Figure 1 · p. 5

5.2 Standard detectors measure a different axis

• Binoculars and Fast-DetectGPT confused some fully delegated and machine-polished reviews, while TMR was sensitive to surface characteristics.EditLens separated these classes, but its AI-editing score does not represent the context-relative external information measured here.Source: §5.2 · pp. 5–6
• As rewriting grew from 11% to 70% of average review length, Self-Conditioning AUC against human reviews rose only from 0.512 to 0.561.Semantic distance stayed below 0.14; the small residual increase suggests very heavy rewriting erodes some measured external information, far less than the surface change alone suggests.Source: §5.2 · p. 6, Table 2 · p. 6

5.3 The score is graded, not binary (RQ2)

• AUC decreased linearly as supplied bullet points increased (r = −0.99); Self-Conditioning moved toward the human regime, unlike standard detectors.The score distributions and AUC trend are shown for Qwen3.5-35B-A3B; EditLens stayed saturated, while Binoculars and Fast-DetectGPT remained strongly machine-like.Source: Figure 2 · p. 6, §5.3 · pp. 6–7, Table 3 · p. 7
• Randomizing which bullet points were supplied preserved the monotone AUC decrease, supporting the role of information amount rather than a fixed disclosure order.The randomized condition showed the same trend as the fixed-order values in Figure 2.Source: §5.3 · pp. 6–7, Figure 2 · p. 6

5.4 Alternative hint families

• Keywords and Surprisal Spans yielded the largest dynamic range; Random Words and Random n-grams kept the trend, while Prefix/Suffix was less sensitive and Top-ranked Sentences stayed near chance.Random Words is the default for its simple local selection, but stochastic averaging adds inference cost.Source: §5.4 · p. 7

6 Edge Cases: Uninformative Human and Surprising LLM Reviews

• A genuinely human low-effort review may score low because it contains little beyond superficial observations, not because it was delegated.Conversely, prompting or decoding can make fully delegated reviews surprising enough to score high without external information.Source: §6 · p. 7
• LazyReview human reviews remained separated from delegated counterparts (1.0 AUC), but a review is labeled lazy if even one sentence exhibits lazy-thinking.The rest of a review labeled lazy may still be informative.Source: §6 · p. 7
• At T = 1.0, discrimination was perfect; at T = 1.5, AUC was 0.926; at T = 2.0, generated reviews scored above human counterparts.The proposed explanation is that increased sampling randomness inflates the residual term I(T; X | C, U), despite I(U; X | C) = 0 for fully delegated reviews.Source: §6 · pp. 7–8
• At T = 2.0, evasion came with degraded coherence, grammar, and logical consistency; such temperatures are atypical and unavailable in some systems.The authors position Self-Conditioning as a screening tool, with low-quality false negatives still identifiable through human inspection.Source: §6 · p. 8

Disclosure budget

• At r∈{0.2, 0.3, 0.4}, older generators remain at or above 0.98 AUC against human reviews, while GPT-5.4 and Gemini-3.1-Pro are more ratio-sensitive.No tested disclosure ratio is uniformly optimal; r remains tunable for newer models.Source: §7 · p. 8
• The disclosure ratio is stable for older generators but remains a tunable component for newer models.All three tested ratios kept older generators at or above 0.98 AUC, whereas GPT-5.4 and Gemini-3.1-Pro were more sensitive to the ratio.Source: §7 · p. 8

Scoring model

• Llama-3.1-70B-Instruct scores Gemini-generated reviews worse than the 8B model, while Qwen3-8B scores Gemini better but GPT-5.4 worse.The 1B variant is weakest overall; the 70B model reaches perfect separation when it matches the generator, but other generators also reach perfect separation.Source: §7 · p. 8

8 Summary and Concluding Remarks

• Self-Conditioning is an unsupervised measure of external information beyond production context, computed with a single open-weights proxy model and requiring no annotated data.Machine-polished reviews remain close to the human score distribution despite substantial rewriting, while conventional ATD methods primarily reflect machine involvement in the observed text.Source: §8 · p. 8
• A low score does not prove LLM generation, and a high score does not establish human authorship; absolute scores depend on the model, context, hints, and text distribution.The authors advise within-setting comparisons and describe Self-Conditioning as decision support for human inspection, not a standalone misconduct detector.Source: §8 · pp. 8–9

A.1 Coding interpretation

• Under an ideal probabilistic code, the information density equals the description length saved when the disclosure is available.The information density is the difference between the description lengths of the text without and with the disclosure.Source: §A.1 · p. 15

A.2 Likelihood-ratio interpretation

• The score’s likelihood ratio tests whether a text–disclosure pair is genuine or whether the disclosure adds no text-specific information beyond baseline context.This gives the criterion a one-shot operational interpretation independent of an asymptotic interpretation.Source: §A.2 · p. 15

A.3 Population interpretation

• At the population level, the expected unnormalised information gain is conditional mutual information; at the instance level, it averages information densities over randomized disclosures.Source: §A.3 · pp. 15–16

A.4 Decomposition into external contribution and residual variation

• The score combines recoverable external information with residual variation from text aspects unpredictable even when context and latent information are known.That residual can reflect stochastic generation or idiosyncratic realization, so a high score need not reflect external contribution alone.Source: §A.4 · p. 16

A.5 Monotonicity under scoring-context augmentation

• I(T; X | C) − I(T; X | C, U) = I(U; X | C) ≥ 0, so adding external information U to the scoring context reduces residual information by the amount recoverable through the disclosure.The claim holds for a fixed target distribution.Source: §A.5 · p. 16
• For sequential contributions, each added item weakly decreases the residual score, yielding I(T; X | C) ≥ I(T; X | C1) ≥ · · · ≥ I(T; X | CK).Each step’s decrease equals the information in the new contribution recoverable through the disclosure.Source: §A.5 · p. 16
• The theorem moves information into the scoring context with the target distribution fixed; the observed trend when varying generation inputs is empirical, not implied by that theorem.Information counts as external only insofar as it is absent from the conditioning context.Source: §A.5 · p. 16

B Additional Figures and Tables

• The machine-polished review’s average token-level boost is 0.505, versus 0.176 for the fully-delegated review.Figure 3 plots token log-probability with and without self-conditioning hints, and their log-likelihood-ratio increment.Source: B Additional Figures and Tables · p. 17, Figure 3 · p. 17
• Human review lengths have a standard deviation of 284.7 words, and LLM-polished reviews inherit the human length distribution; length alone does not explain score distributions.GPT-5.4 is more verbose than the human baseline, whereas Gemini-3.1-Pro tends toward shorter reviews.Source: B Additional Figures and Tables · p. 17, Table 4 · p. 18
• Figure 4 displays LazyReview scores alongside generated counterparts, while Figure 5 shows scores and AUCs for high-temperature generated counterparts against human reviews.Source: B Additional Figures and Tables · pp. 17–18, Figure 4 · p. 17, Figure 5 · p. 19

C Detailed List of Models Used

• The benchmark uses pre-2022 OpenReview human reviews and generated counterparts; additional review regimes include LLM-polished texts and reviews generated from structured human-review bullet points.The appendix lists the models and generation settings used for these data.Source: C Detailed List of Models Used · p. 20

D Parsing PDFs

• Matching reviews to papers required converting OpenReview PDFs to Markdown; papers whose figures caused parsing crashes and their reviews were excluded.Source: D Parsing PDFs · p. 20

E Baseline Details

• The baselines include supervised ModernBERT and TMR detectors, unsupervised Binoculars and Fast-DetectGPT, and EditLens, which estimates the extent of AI-assisted editing.Binoculars and Fast-DetectGPT use Falcon-7b and its instruct version in the reported configurations.Source: E Baseline Details · p. 21

• EditLens assigns polished versions AI-editing scores between 0.4 and 1, consistent with its training to estimate how much they differ from their originals.Figure 10 shows score densities for fully generated reviews and polished human reviews.Source: §E.1 · p. 21, Figure 10 · p. 22
• Unsupervised detectors tend to confound polished reviews with Claude-Sonnet-3.5 and GPT-4o generations, while ModernBERT shows poor out-of-domain capabilities.Source: §E.1 · p. 21

F More Results for Self-Conditioning

• Low TPR@0.1%FPR for some recent generator models illustrates limits of the detector; Table 7 reports fixed-FPR results across generators.Source: F More Results for Self-Conditioning · p. 22, Table 7 · p. 23

G.1 Random-ordered bullet points

• The decreasing AUC trend persists when supplied bullet points are sampled uniformly rather than in a fixed order.Figure 11 shows score distributions and AUC against human texts.Source: §G.1 · p. 23, Figure 11 · p. 23

G.2 Other Generators for Hybrid Reviews

• Across Mistral and Gemma generations, scores move closer to the human distribution as more bullet-point information is supplied, though Mistral remains far from it while Gemma becomes very close.Figures 12 and 13 show score densities and AUC against human scores.Source: §G.2 · pp. 23–24, Figure 12 · p. 24, Figure 13 · p. 24

G.3 Other Scoring Models in Peer-review Settings

• Table 6 reports AUC values for alternative scoring models in the peer-review setting.Source: §G.3 · p. 24

H NeurIPS & Other Years

• The study applies Self-Conditioning to ICLR 2020, ICLR 2021, and NeurIPS 2021 subsets; the possible positive-review bias in NeurIPS does not seem to affect the score distribution.Figures 14–16 show score distributions and AUC values for these subsets.Source: H NeurIPS & Other Years · p. 24, Figure 14 · p. 25, Figure 15 · p. 25

I High Temperatures Reviews

• Reviewers praise Riemannian subgradient descent for orthogonal dictionary learning, its theoretical guarantees, and tools for analyzing nonsmooth nonconvex optimization.They also describe the approach as computationally efficient and supported by synthetic and real-image experiments.Source: I High Temperatures Reviews · pp. 26–27
• Reviewers flag unresolved extensions to overcomplete dictionaries and an n^4 term in the best sample-complexity analysis, while one questions whether complete-dictionary insights generalize.One reviewer also calls the extension to the nonsmooth case the main challenge and questions the broader utility of complete dictionary learning.Source: I High Temperatures Reviews · pp. 26–27

J.1 Peer Review Prompts

• For fully generated ICLR 2019 reviews, GPT-5.4 and Gemini-3.1-Pro received identical system/user prompts, reviewer guidelines, a review template, and the human decision to produce reviews as close as possible to human ones.GPT-5.4 used temperature 0.9 and max_completion_tokens=2048; Gemini-3.1-Pro used the same generation parameters.Source: §J.1 · p. 29
• For controlled variants, Llama and Qwen rewrote human reviews without changing technical content or sentiment, while Llama-3.1-8B-Instruct summarized reviews into eight bullets for partial-information generation.The bullet categories covered contributions, strengths, weaknesses, evidence quality, novelty, clarity, missing experiments, and final judgment.Source: §J.1 · pp. 30–31
• The AI-review prompt directs the model to treat partial human-review arguments as its foundation, expand them using the manuscript, and align its assessment with conference guidance.It also requires a complete review without internal reasoning or preamble.Source: §J.1 · pp. 31–32

J.2 Graded Rewriting Prompts

• Rewriting prompts range from light typo and grammar fixes that preserve voice and structure to major overhauls for polished academic critique.Moderate and heavy settings progressively improve flow or substantially rewrite the text while maintaining its meaning or professional tone.Source: §J.2 · p. 32

K Further Details About Hint Selection

• The estimator compares target-review token log-probabilities under the baseline context and the same context augmented with a selected hint; r controls the disclosure budget.Hints are prepended as “Hints: {x}”; random words, prefix/suffix disclosure, and n-grams provide alternatives with differing coherence and formatting dependence.Source: K Further Details About Hint Selection · pp. 32–33
• Other hint methods select high-TF–IDF keywords or sentences, while surprisal spans reveal high-surprisal regions under the baseline model.Keyword and sentence ranking use corpus-fitted TF–IDF; sentence selection retains original order, and surprisal spans greedily select high-scoring non-overlapping windows.Source: K Further Details About Hint Selection · p. 33

L.1 Theoretical Complexity

• A deterministic single-hint score requires two scoring-model forward passes; averaging N hint realizations requires 1+N passes.For a standard Transformer, per-pass cost scales roughly quadratically with sequence length, while hint extraction is generally cheap relative to model scoring.Source: §L.1 · p. 34

L.2 Hardware Used

• Experiments used NVIDIA H100 GPUs with 80 GB memory and bfloat16 precision, with approximately 4000 GPU-hours of project compute excluding proprietary API generations.Most compute was spent on proxy-model scoring passes.Source: §L.2 · p. 34

M Additional Results For All Hint Selection Methods

• Ten supplementary tables report Qwen3.5 peer-review AUCs across hint-selection methods, but the supplied captions do not provide values or identify a winning method.The tables cover random words, prefix and suffix disclosure, keyword extraction, n-grams, surprisal spans, and top-ranked sentences.Source: Table 8 · p. 35, Table 9 · p. 35, Table 10 · p. 35

N Licenses

• Table 18 lists licenses and terms of use for external models, datasets, and baseline assets, reporting service terms for proprietary API models.Source: Table 18 · p. 38