PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-critiquing models for assisting human evaluators

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike

arXiv:2206.05802v2cs.CLcs.LG

TL;DR

The paper addresses how to provide scalable feedback on outputs that are difficult for humans to evaluate. It trains models to generate and assess critiques, finding that larger models produce more helpful critiques and can use them to improve their outputs.

  • Problem

    The paper studies how to provide effective feedback on model outputs when correctness is difficult and costly for humans to evaluate.

    Source: §1.1 · p. 1

  • Method

    The authors train models on base-task and critique demonstrations, then compare critique generation with binary discrimination tasks for answers and critiques.

    Source: §2 · p. 3, §2.1 · p. 3

  • Results

    Larger models produce critiques rated as more helpful by humans and improve outputs through critique-conditional refinement.

    Source: §4 · p. 9

  • Takeaways & Limitations

    The results provide a proof of concept for using AI-generated critiques to assist feedback on tasks that are difficult for humans to evaluate directly.

    Source: §7 · p. 18

    The topic-based summarization task lacks a robust or objective process for validating answer and critique quality.

    Source: §7.2 · p. 18

Abstract

from arXiv · show

We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own self-critiques as feedback, refining their own summaries into better ones. Finally, we motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability. Our measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results are a proof of concept for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. We release our training datasets, as well as samples from our critique assistance experiments.

1 Introduction

Evaluating difficult model outputs requires substantial human expertise, motivating AI-assisted oversight through natural-language critiques. The paper shows that critiques improve human flaw-finding, scale with model capability, enable answer refinement, and reveal gaps between generation, discrimination, and critique abilities.

1.1 Motivation

  • Because correctness and factual veracity are costly and expertise-intensive to evaluate, scalable oversight asks how humans can provide feedback on difficult tasks.Human-feedback methods fundamentally depend on people’s ability to demonstrate and evaluate model-output quality.Source: §1.1 · p. 1

1.2 Contributions

  • Assisted labelers found about 50% more critiques in model- or human-written answers, and found intended critiques 50% more often in deliberately misleading answers.Most critiques found with assistance came directly from model critiques, while the number of used model critiques was comparable to critiques found without assistance.Source: Figure 1 · p. 2, §1.2 · p. 2

1.1 Motivation

  • Natural-language critiques are proposed as assistance that helps evaluators notice misleading flaws, provided critique models can explain flaws they recognize.The work fine-tunes language models jointly on summarization and critique tasks, primarily finding errors in topic-based summaries.Source: §1.1 · p. 2, §1.2 · p. 2

1.2 Contributions

  • Larger models generally critique themselves better despite producing harder-to-critique answers, while synthetic-task trends are similar but less consistent.Critique helpfulness scales favorably with model capabilities, keeping pace with models’ ability to produce more convincing answers.Source: §1.2 · p. 2
  • Large models improve their own answers using generated critiques, with rejection-sampled critiques outperforming direct refinement and improvement scaling favorably with model size.Small models showed no improvement for either refinement approach.Source: §1.2 · p. 2
  • The paper proposes generator-discriminator-critique gaps and finds no clear trend of critique performance catching up to discrimination performance.This implies larger models still possess relevant knowledge they do not articulate as critiques.Source: §1.2 · p. 3
  • The authors release training datasets and assistance-experiment samples, including misleading answers and their intended flaws.The training release includes human-written critiques, refinements, and critique evaluations.Source: §1.2 · p. 3

2 Dataset collection and model training

The paper trains models jointly on base tasks, critiques, discrimination judgments, and refinement through iterative data collection and supervised fine-tuning. Experiments use topic-based summarization and synthetic tasks, with critiqueability, helpfulness, and model scale defined for evaluation and comparison.

2 Dataset collection and model training

  • The training pipeline collects base-task demonstrations, then critique demonstrations, and iteratively fine-tunes models jointly across tasks and rounds.Each round adds demonstrations while training on all data collected so far.Source: §2 · p. 3

2.1 Structure of tasks

  • The framework defines critiqueability as detecting flaws in answers, helpfulness as detecting legitimate shortcomings in critiques, and conditional refinement as improving an answer using a critique.These tasks are defined alongside the arbitrary question-and-answer base task.Source: §2.1 · p. 3

2.2 Topic-based summarization

  • The main base task is topic-based summarization, supported by over 6,000 topical queries and summaries across more than 2,000 passages.Summaries target 2–10 sentences and focus on a specific aspect of a passage rather than the whole text.Source: §2.2 · p. 4

2.2.1 Data collection

  • Demonstrations use human-generated questions and may use model- or human-generated inputs, with later rounds relying on model answers and critiques for critique and helpfulness tasks.Data collection begins with base-task demonstrations, then adds critiqueability, critique, refinement, and helpfulness demonstrations.Source: §2.2.1 · p. 4

2.2.2 Models

  • Pre-trained transformer decoders are fine-tuned by supervised learning on human labels across all tasks, with joint training intended to avoid capability asymmetry between base and critique models.Training masks non-demonstration input tokens and uses one epoch with learning rate tuning.Source: §2.2.2 · p. 5

Critiqueability and helpfulness score

  • Critiqueability and helpfulness scores are computed from the model’s binary-answer logits as Pr[Yes] / (Pr[Yes] + Pr[No]).The critiqueability score can determine whether to request a critique, while helpfulness can assess it afterward.Source: Critiqueability and helpfulness score · p. 5

Model scale

  • Model scale is measured by supervised fine-tuning loss in nats per token rather than parameter count, because directly comparable models show a strong loss–parameter correlation.This loss-based scale is used for scaling laws across models with different architectures.Source: Model scale · p. 5

2.3 Synthetic tasks

  • Four synthetic tasks use binary ground-truth oracles instead of human data collection, and are chosen so critique evaluation is easier while the base task remains difficult but feasible.Synthetic training uses generated oracle demonstrations followed by critiqueability and critique data; unlike summarization, it varies datasets by model size and omits a helpfulness-training round.Source: §2.3 · pp. 5–6

3 Assisting critique finding

Model-generated critiques help human labelers find more and better-supported flaws across model, honest human, and misleading human summaries, without appreciably increasing task time. The experiments also release a comprehensive dataset of assistance, critiques, ratings, and intended critiques.

3.1 Motivation

  • The study tests critique assistance because finding critiques is central to answer evaluation and suggesting critiques is a natural form of assistance.The authors also considered it the easiest task for measuring assistance effects after finding comparison-based evaluation more difficult.Source: §3.1 · p. 6

3.2 Setup

  • Labelers critique long topic-based summaries with or without eight temperature-varied model critiques, evaluated on model, honest human, and deliberately misleading human answers.Six labelers cover six conditions, and labelers report answer quality, critique severity, and critique category.Source: §3.2 · p. 7

3.3 Findings

  • Assistance helps labelers find more critiques across all answer conditions, increasing critiques across severity levels and categories while leaving task time largely unchanged.The largest gains are for minor and coverage critiques, but major, critical, and accuracy critiques also increase.Source: Figure 2 · p. 7, §3.3 · p. 8
  • Assistance makes answers more frequently critiqued and lowers ratings; for model answers, critique coverage rises from 54% to 78% and average ratings fall from 5.6 to 4.9.Most used critiques are accepted with little editing: only 15.3% of used critiques were edited further.Source: §3.3 · p. 8
  • The critique model was trained on the same labeler pool, effectively bootstrapping those labelers to be more careful at the base task.Because labelers see many suggestions, the experiment may sometimes accept misleading critiques, motivating the intended-critique validity check.Source: §3.3 · p. 8, §3.3.1 · p. 8

3.3.1 Checking for intended critiques

  • With assistance, labelers find intended critiques 45 ± 5% of the time versus 27 ± 5% without assistance, while valid-critique rates remain 71% versus 73%.The number of valid critiques is 0.96 ± 0.09 versus 0.62 ± 0.07, and valid novel critiques are 0.24±0.06 versus 0.18±0.05.Source: §3.3.1 · p. 8

3.4 Dataset release

  • The authors release a comprehensive dataset containing assistance, used and written critiques, ratings, and intended critiques, with random samples in Appendix F.2.Source: §3.4 · p. 8

4 Critique quality results

Critique quality improves with model scale: larger models produce more helpful critiques and self-critiques, while critique-conditional refinement can improve outputs. However, model critiques remain less helpful than human critiques, and their refinement advantage is not consistently compute-competitive.

4.1 Helpfulness

  • Human labelers assess critique helpfulness and whether critiques are clearly and unambiguously helpful, checking that supervised fine-tuning does not produce overly nit-picky critiques.The evaluation focuses on answers for which humans identified critiques as possible.Source: §4.1 · p. 9
  • Our model’s critiques are more helpful than InstructGPT baselines but substantially less helpful than human critiques.The evaluation compares supervised fine-tuned critiques with zero-shot and few-shot InstructGPT-style baselines on critiqueable answers.Source: §4.1 · p. 9, Figure 3 · p. 9

4.2 Self-critiquing helpfulness and scaling

  • Larger models are better at self-critiquing despite producing answers that are harder to critique, with this pattern holding without filtering for critiqueable answers.On synthetic tasks, similar critiqueable-case trends appear, but the overall scaling story is less clear and is not established as fundamental.Source: Figure 4 · p. 10, §4.2 · p. 11

4.3 Refinements

  • Critique-conditional refinement uses both an answer and its critique, whereas direct refinement uses only the answer, enabling a direct test of whether critiques help.The experiments also select critiques or refinements with best-of-N scoring and compare refined outputs against the original using critiqueability scores.Source: §4.3 · p. 11

4.3.1 Findings

  • Conditional refinements outperform direct refinements when critiques are selected by helpfulness, with larger N improving results, while direct-refinement rejection sampling remains stronger.Both refinement methods significantly outperform original outputs for larger models but have little to no effect for smaller models.Source: §4.3.1 · p. 12

5 Generator-discriminator-critique (GDC) gaps

The GDC framework compares generation, discrimination, and critiquing through GD, GC, and CD gaps. Results show consistently positive GD and GC gaps, while CD gaps are task-specific and do not consistently close with scale.

5.1 Setup

  • The framework defines G as generation, D as discrimination, and C as critiquing, then measures XY gaps as how much Y performance exceeds X performance.GC captures self-critiquing, GD captures recognizing poor outputs, and CD captures giving human-understandable critiques for flaws the model recognizes.Source: §5.1 · p. 12, §5.2 · p. 14

5.2 Measuring gaps

  • For C, the model generates a critique and an answer is judged poor when a human validates any critique as valid and severe.Primary results use best-of-N with N = 2; C is a convenient, non-canonical definition that treats critiques as a discriminator.Source: §5.2 · pp. 14, 16

5.2.1 Sample quality

  • Sample quality is measured using Likert ratings, binary uncritiqueability, or win rate against a fixed generator distribution, but topic-summary labels may fail systematically.For synthetic tasks, uncritiqueability corresponds to oracle accuracy; win rate makes comparisons across model sizes natural.Source: §5.2.1 · p. 16

5.2.2 Critique severity

  • Critique severity uses human severity for summarization and binary helpfulness ground truth for synthetic tasks, allowing C to exploit limited search unavailable to D.This search effect could produce a negative CD gap, but should diminish at scale because critique validation is easier than finding flaws.Source: §5.2.2 · p. 16

5.3 Results

  • Positive GD and GC gaps consistently appear, while CD gaps vary by task and do not show a general trend toward closing with scale.Topic summarization has an approximately constant positive CD gap; synthetic-task trends vary, with RACE opposite to most tasks.Source: Figure 8 · p. 14, §5.3 · p. 17
  • The CD gap may be harder to close on difficult realistic tasks because discriminators can exploit answer-writer style or labeler-specific critiqueability without identifying actual flaws.This confound is described for topic-based summarization but does not occur in the synthetic tasks.Source: §5.3 · p. 17

6 Related work

Prior work spans scalable alignment experiments, natural-language assistance for human data creation and evaluation, critique datasets and models, and natural-language feedback for refining model outputs. This work differs by focusing on model-generated critiques that assist humans and improve models’ own summaries.

Scalable alignment experiments

  • Scalable alignment studies have explored iterative amplification, debate, recursive reward modeling, and debate-like experiments on algorithmic, toy, and realistic tasks, with generally mixed results in human-debater settings.[CSA18], [ICA18], [PTP+22], [AZWG21], and [WOZ+21] investigate these approaches across different task and evaluator configurations.Source: §6 · p. 17

Human assistance with natural language

Critique datasets and models

  • Existing critique resources include datasets of factual claims with evidence, peer reviews, social-media disagreements, and story critiques.The datasets were introduced by [TVCM18], [KAD+18],,, and [MST+21].Source: §6 · p. 17
  • Project Debater trains models to conduct free-text debates and rebut arguments, whereas this work focuses on debating against models rather than humans.The distinction concerns the target of the debate: Project Debater engages with humans, while this work studies model-focused critique assistance.Source: §6 · p. 17

Natural language refinements

7 Discussion … A.3 Base tasks

The discussion presents AI-assisted critique as a promising path toward scalable oversight and natural-language feedback, while emphasizing limitations around ground truth, articulability, task difficulty, and setup. The appendices describe the labelers, data-collection procedures, critique and helpfulness tasks, and topic-based summarization design.

7 Discussion

  • AI assistance produced significant gains in human evaluation, supporting feedback assistance as a proof of concept for scalable oversight.The experiments found gains even though topic-based summarization was not especially difficult for human labelers.Source: §7 · p. 18

7.1 Implications for alignment research

  • Large language models can already critique meaningfully, improve at self-evaluation with scale, and support alignment methods such as reward modeling [LKE+18] and debate.InstructGPT models critique well few-shot and zero-shot, while Figure 3 and Figure 4 provide the cited evidence for these capabilities and scaling trends.Source: §7.1 · p. 18
  • The work proposes generator-discriminator-critique gaps as measures of alignment-relevant capabilities and training-signal quality, including in smaller synthetic domains.Increasing critique performance relative to generator and discriminator performance is identified as an under-explored route toward better-aligned models.Source: §7.1 · p. 18
  • Models can condition on natural-language critiques to improve their own outputs, making human natural-language feedback a promising training direction [SCC+22].The paper connects this result to preference-learning feedback [CLB+17] and related chain-of-thought findings [WWS+22b].Source: §7.1 · p. 18

7.2 Limitations

  • The main limitations are missing objective ground truth, potentially mistaken or misleading evaluations, unarticulated model knowledge, subjective preferences, and tasks where evaluation is not simpler than generation.Agreement on critique helpfulness rankings was around 75%, and the authors warn that harder tasks may make labelers increasingly unreliable.Source: §7.2 · p. 18

7.3 Future directions

  • Future work includes studying misleading models, reducing discriminator-critique gaps, recursive reward modeling, alternative assistance methods, iterative refinement, disagreement reconciliation, and natural-language training.The authors also seek more difficult tasks with more objective ground truth, including coding, mathematics, riddles, and book-length question-answering.Source: §7.3 · p. 19

Appendix; A.1 Labelers; A.2 Collection details

  • The appendices document hourly OpenAI-contracted English-fluent labelers, occasional quality checks, and three data-collection rounds covering base, critique, and helpfulness tasks.The collection process used distinct web interfaces and shared-task agreement checks.Source: §A.1 · p. 25, §A.2 · p. 25

A.2.1 Base task; A.2.2 Critique task; A.2.3 Helpfulness task; A.3 Base tasks

  • The dataset uses topic-based summarization tasks designed to require information beyond keyword matching, with structured misleading answers, critiques, refinements, severity and category labels, quote highlights, and helpfulness judgments.The appendix also records the task variants, critiqueability assumptions, model-critique generation, and the fact that most reported results use topic-based summarization while earlier question-answering and summarization tasks remain in training data.Source: §A.2.1 · p. 25, §A.2.2 · pp. 25–26

A.4 Auxiliary tasks … B.1 Theory background

The final training data included auxiliary question, corroboration, and quote-retrieval tasks, with task conversions used for augmentation. The paper frames critique-based assistance through an analogy to interactive proofs and complexity classes.

A.4 Auxiliary tasks

  • Final models included question creation, corroborations, corroboration quotes, and question quotes as auxiliary supervised-training tasks.Quote retrieval was conditioned either on the answer span or only on the question.Source: §A.4 · p. 27
  • During training, conditional refinement became direct refinement 50% of the time, while corroboration quotes became question quotes 50% of the time.Source: §A.4 · p. 27
  • The final setup used single critiques rather than critique slates, simplifying assistance despite potentially reducing duplicate critiques.Source: §A.4 · p. 27

A.5 Formatting details

  • Examples were formatted as passage, question, answer, critiqueability, critique, helpfulness, and refinement, enabling critique and critiqueability evaluation together.Critique inputs begin with the “Yes” critiqueability token; task-specific formats omit or skip fields as needed.Source: §A.5 · p. 28
  • The authors found little effect from reordering in synthetic RACE experiments and left task-dependent format optimization for future work.Source: §A.5 · p. 28

B Complexity theory analogy

  • The complexity-theory analogy treats humans as fixed-computation agents and models as potentially computationally unbounded agents interacting to solve decision problems.This motivates comparing scalable oversight with interactive proof systems.Source: B Complexity theory analogy · p. 28, §B.1 · p. 28

B.1 Theory background

  • Behavioral cloning corresponds to P, human-evaluated training to NP/co-NP, and debate schemes to PSPACE, with n-step variants matching polynomial-hierarchy levels.Source: §B.1 · p. 28
  • Cross-checking a second prover can prevent deceptive strategies by a single prover in both interactive-proof settings and scalable oversight.Source: §B.1 · p. 28

B.2 Proof systems in practice · C GDC gaps: discussion and extra results · C.1 Informal intuition

The paper connects recursive human-assisted verification to higher levels of the polynomial hierarchy, while discussing limits as models improve. It also characterizes gaps among generating, discriminating, and critiquing abilities, whose relationships depend on task structure.

B.2 Proof systems in practice

  • RLHP empirically reaches NP/co-NP by having models find solutions while humans only verify them [CLB+17] [BJN+22] [MTM+22].Source: §B.2 · p. 28
  • n-step recursive reward modeling corresponds to the nth polynomial-hierarchy level, with 2-step RRM analogous to access to an NP/co-NP oracle.Source: §B.2 · p. 29
  • Moving up the hierarchy is inherently critique-like: ΣP_2 asks humans to recognize critiques, while ΣP_3 requires an assistant trained to critique critiques.Source: §B.2 · p. 29
  • Iterative refinement may be powerful when human computation is not limiting, but its ability to solve problems outside P is unclear and its usefulness may decline as models improve.Source: §B.2 · p. 29

C GDC gaps: discussion and extra results

C.1 Informal intuition

  • The GD gap typically reflects G ≤ D: writing good answers is harder than recognizing flawed ones, though task structure and rejection sampling can reduce the gap.Source: §C.1 · p. 29
  • The CD gap reflects C ≤ D because recognizing flaws can be easier than articulating them, while the GC relationship is ambiguous and task-dependent.Examples include intuitive cat–dog discrimination without explanation, photogenic smiling without critiquing inauthentic smiles, and critiquing poor plots more easily than writing good ones.Source: §C.1 · p. 29

C.2 Relevance to model training and scalable oversight … C.3.1 Reasons to expect negative CD gaps

The section argues that critiqueability should be pushed as far as possible for scalable oversight, while distinguishing critique-based training signals from generation and discrimination. It also explains how measurement choices can produce negative CD gaps and discusses alternatives when ground truth is unavailable.

C.2 Relevance to model training and scalable oversight

  • Training against critiqueability can elicit misleading outputs when models cannot explain why worse outputs are worse, so critique models should push the GC gap positive (C ≥G).Otherwise, optimizing critiqueability may favor outputs that look good to humans but are not actually good.Source: §C.2 · pp. 29–30
  • Closing the CD gap would help humans identify all poor or misleading outputs, offer stronger assurance than closing GC, and make critique-based oversight more comparable to discrimination.The authors also warn that flawed critiqueability labels could train a discriminator—and then a rejection-sampled generator—to favor misleading outputs.Source: §C.2 · p. 30
  • For difficult tasks without ground truth, critique models may outperform discriminators, although critique gaps and the risk of unexplainable vulnerabilities remain unknown.A helpfulness model might identify an undiscovered security vulnerability that human labels and a discriminator would miss, even though correct labels would be preferable when possible.Source: §C.2 · p. 30

Relaxing ground truth

  • When ground truth is unavailable, more knowledgeable humans can provide D labels while less knowledgeable humans provide C labels with assistance from a model trained on their judgments.Capability disparities can be created through restricted time budgets or by hiring less-expert labelers, following the setting introduced by.Source: Relaxing ground truth · p. 30

Process versus outcomes

  • Process-based systems may merit investment even when outcome-based systems perform better, because proxy outcomes are tempting for costly long-horizon tasks and reveal little about generalization.Closing CD frames human-understandable process as potentially competitive with outcome-based training, paralleling [SB22].Source: Process versus outcomes · p. 30

C.3 Measuring gaps discussion

  • The gap measures define G by average generator-sample performance, D by best-of-N selection against a discriminator, and C by best-of-N selection against critique severity.These definitions motivate the later discussion of why C can differ from D.Source: §C.3 · p. 31

C.3.1 Reasons to expect negative CD gaps

  • Cm makes C more analogous to D by selecting outputs using a model’s predicted critique helpfulness; Cm and C should converge for very large models, while D ≤C may persist.The proposed reasons include vocalized critiques helping discrimination as chain of thought [WWS+22b] [WWS+22a] and unequal compute, such as searching over critiques.Source: §C.3.1 · p. 31
  • Negative CD gaps appeared on Addition, Alphabetize, and RACE, likely because C uses an oracle; the authors suspect Cm < D but do not investigate further.C directly searches against human-checked critique validity, whereas Cm uses a helpfulness model as a discriminator.Source: §C.3.1 · p. 31

C.3.2 Alternative C definitions · C.3.3 Evaluating reward signal without training

Alternative critique definitions show that best-of-8 critiques improve human evaluation, although reviewing multiple critiques still trails discriminator ability, while model-only variants are noisy. GD and GC gaps provide training-free tests of reward-learning and critique-based signals, with robustness limited by over-optimization and imperfect discriminators.

C.3.2 Alternative C definitions

  • Figure 11 suggests that generating many critiques and selecting by helpfulness can improve helpfulness discrimination for large models, although the model-only results remain noisy.The idealized limit M →∞ is expected to catch up to or exceed discriminator performance.Source: Figure 11 · p. 32, §C.3.2 · p. 32

C.3.3 Evaluating reward signal without training

  • Best-of-N gains improve slightly with model size for GD, whereas GC results are noisy: best-of-4 beats best-of-2, best-of-8 may underperform best-of-4, and gains do not grow with model size.Figure 12 reports GD scaling, while Figure 13 reports GC scaling and finds critique models are not robust discriminators.Source: Figure 12 · p. 32, Figure 13 · p. 33

C.3.2 Alternative C definitions

  • Ch8 shows humans a slate of eight critiques and selects answers using the validity and severity of the most critical critique.This extends human assistance by allowing review of multiple pieces of critique assistance.Source: §C.3.2 · p. 32
  • C8 selects one best-of-8 critique by helpfulness score, then has humans judge the answer using validity and severity.C8 is the same basic procedure as C but supplies a better critique.Source: §C.3.2 · pp. 31–32
  • Using a best-of-8 critique improves evaluation over C, while humans reviewing eight critiques outperform humans reviewing one optimized critique but still trail discriminator ability.These variants are evaluated in Figures 10, with C8 > C and Ch8 > C8.Source: §C.3.2 · p. 32
  • Model-only results are noisy: Cm is competitive with C, but increasing the number of critiques does not clearly help, with Cm versus Cm8 inconclusive.Cm8 removes humans from the loop and relies on helpfulness scores being comparable across answers.Source: §C.3.2 · p. 32

C.3.3 Evaluating reward signal without training

  • GD and GC gaps offer training-free tests of reward learning and critique-based training signals by measuring best-of-N performance without actually training with reinforcement learning.GD uses a discriminator or preference-based reward model, while GC uses human checking of critiques as a discriminator.Source: §C.3.3 · p. 33
  • GD and GC scaling with N test in-distribution robustness, but negative gaps can arise when discriminators perform worse than chance, including through generator over-optimization.The over-optimization mechanism is discussed in relation to [MG18].Source: §C.3.3 · p. 33

D 2-step debate … E.2 Quotes as assistance

In 2-step debate, best-of-N helpfulness-based discrimination remained worse than direct critiqueability, although larger models or better severity modeling could change the result. Additional assistance experiments found reliable benefits from human-written critiques, but model-written critiques and quote highlighting offered limited gains.

D 2-step debate

  • The debate test compared critiqueability discrimination against helpfulness scores from best-of-N critiques on paired misleading and honest answers.This design avoided relying on humans to find critiques when establishing ground truth.Source: D 2-step debate · p. 33
  • Best-of-N helpfulness discrimination improved with N but remained worse than direct critiqueability, including as N →∞, suggesting naive 2-step debate underperformed RLHP.The comparison used misleading versus honest answers and the largest model with temperature 0.5-sampled critiques.Source: D 2-step debate · pp. 33–34, Figure 14 · p. 34
  • The negative debate result may reflect helpfulness models favoring nit-picky over severe critiques, motivating severity models or critique-conditional preference-based reward models.Debate may also become more useful with larger models, less biased helpfulness-based training, or domains where critiqueability labels are harder to obtain.Source: D 2-step debate · p. 34
  • Debate may initially be harder to implement than recursive reward modeling because it requires a robust helpfulness model, whereas humans can interact with multiple critiques directly.Synthetic 2QBF-like tasks could be informative when helpfulness is easy to learn but critiqueability is hard.Source: D 2-step debate · p. 34

E.1 Assistance for comparisons

  • Comparison assistance also improved labeling speed, but the experiment required substantial additional effort and labeling and was discontinued.The setup aimed to compare assisted single-human accuracy with an unassisted human ensemble as ground truth.Source: §E.1 · pp. 34–35, Figure 15 · p. 35
  • Human-written critiques significantly improved single-human comparison accuracy against ensembles of five humans, while model-written critiques produced improvements within noise.Assistance increased the fraction of answers with critiques found but decreased Likert-score judgments.Source: §E.1 · pp. 34–35, Figure 16 · p. 35

E.2 Quotes as assistance

  • The quotes experiment retrained models to generate supporting excerpts that labelers could reveal and use to navigate to corresponding text locations.Quotes were drawn from the response and/or source text.Source: §E.2 · p. 35
  • Quotes did not increase critiques found, saved labelers a little under a minute, and performed similarly to longest-common-substring highlighting.The baseline saved almost the same amount of time without generated quotes.Source: §E.2 · p. 35

E.3 Ablation of number of critiques … F.2 Assistance samples

Using eight rather than four model-generated critiques increased the number of critiques found, while the appendix provides randomized self-critique and assistance samples with their collection procedures. These samples include model outputs, critiqueability and helpfulness assessments, human helpfulness labels, and released datasets.

E.3 Ablation of number of critiques

  • Eight critiques produced nearly twice the increase in critiques found as four, although four may have made critique-finding faster than no assistance.The four-critique condition nevertheless found fewer critiques than the eight-critique condition.Source: §E.3 · pp. 35–36

F Samples

  • F Samples uses uniformly random sampling subject to stated constraints and omits samples with triggering content.Triggering content occurred only once.Source: F Samples · p. 36

F.1 Self-critique and helpfulness samples

  • The self-critique samples include model answers, critiqueability scores, self-critiques, self-assessed helpfulness scores, and human helpfulness judgments.These samples are presented in Tables 6-15.Source: §F.1 · p. 36
  • Self-critique samples come from the largest model, use random non-zero answer temperature and matching critique temperature, and draw eight deduplicated critiques per answer.The sampling procedure mirrors deduplication used in the assistance experiments.Source: §F.1 · p. 36

F.2 Assistance samples

  • The assistance samples randomly select experiment 3.4 examples while requiring every assistance condition to contribute at least one critique used.This condition ensures that all assistance settings are represented.Source: §F.2 · p. 36
  • Assistance samples appear in Tables 16-25, and the complete sample dataset is released through the assistance dataset release.The release extends beyond the displayed examples.Source: §F.2 · p. 36

F.3 Refinement samples

This section presents self-critique and refinement samples from the largest model across diverse summarization examples, including random-critique, best-of-8, and direct refinements. It also illustrates critiques and assisted-human evaluations that identify specific omissions in answers, such as missing emotional, setting, event, factual, argumentative, and descriptive details.

Refinement samples

  • The samples include three refinements—random-critique, best-of-8-critique, and direct—and retain cases where human labeling ranked all refinements above the original.The samples are presented in Tables 26–35.Source: §F.3 · p. 36

Helpfulness experiments

  • In helpfulness examples, critiques identify missing domestic-violence statistics and unnecessary or misplaced content in answers about St. Mark’s Church.One critique notes that domestic violence accounts for 80% of U.S. hostage situations, while another flags an unnecessary final sentence and a paragraph-ordering issue.Source: §F.3 · pp. 39–40

Assistance experiments

  • For the Mike-disappearance example, assisted human critiques identify the false claim that Mike hid in the woods to urinate and the omission that he was dead; model critique helpfulness scores are 0.71 and 0.51.A model critique with helpfulness score 0.42 corrects the setting from a court to a golf course.Source: §F.3 · p. 42
  • For the elevator example, assisted critiques emphasize that the blonde woman is an old acquaintance and that the answer omits their prior relationship and conversation details.Model helpfulness scores are 0.59 for the missing acquaintance relationship, 0.84 for omitted interaction details, and 0.64 for their former closeness.Source: §F.3 · p. 46
  • For the Isle of Man Treasury example, critiques flag omitted identification of the Treasury, confusion with the Customs and Excise Division, and irrelevant staffing information.The model assigns helpfulness scores of 0.44 to removing the staffing detail and 0.17 to adding that the Treasury is the government’s finance department.Source: §F.3 · p. 49

Conditional refinement

  • The refinement samples span sports, literary, news, political, and descriptive prompts, indicating that critique targets vary with the source and question.Examples include the 1997 Singer Akai Cup, literary stories, UK car production, climate denial, protests in Washington, and a doctor’s appearance.Source: §F.3 · pp. 51, 53, 54
  • Refinement critiques identify omitted details such as positive emotions, a wife’s emotional effect, a video-game setting, and Celeste falling on the stairs.These examples show critiques correcting incomplete answers by targeting details absent from the original response.Source: §F.3 · pp. 51–52
  • Several critiques preserve or request concrete source-specific information rather than merely judging an answer as generally incomplete.The requested additions include a video-game setting, falling on stairs, education-influence tactics, and smiling.Source: §F.3 · pp. 51–54
  • Critiques also flag missing factual context, including tactics used by the climate-denier movement to influence education and the doctor’s smile.The examples concern both an article’s substantive argument and a story character’s appearance.Source: §F.3 · pp. 53–54

Related papers

Loading 2206.05802v2…