PaperlayerView exact revision on arXiv

Source-linked AI summary

Self-critiquing models for assisting human evaluators

William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, Jan Leike

arXiv:2206.05802v2cs.CLcs.LG

TL;DR

The paper addresses how to provide feedback on outputs when humans cannot efficiently evaluate correctness. It fine-tunes language models to generate critiques and finds that larger models produce more helpful critiques, improve outputs through critique-based refinement, and can assist human oversight.

  • Problem

    Scalable oversight asks how to provide effective feedback on tasks whose correctness requires substantial human effort and expertise.

  • Method

    The authors fine-tune language models with behavioral cloning to generate natural-language critiques and introduce measures comparing generation, discrimination, and critiquing ability.

  • Results

    Larger models write more helpful critiques and can improve outputs through critique-conditional refinement, while positive GD and GC gaps are consistently observed except when models cannot perform the task.

  • Takeaways & Limitations

    The results provide a proof of concept for AI-assisted human feedback as an approach to scalable oversight, even for topic-based summarization.

  • Takeaways & Limitations

    The topic-based summarization task lacks a robust or objective process for validating the quality of answers or critiques.

Abstract

from arXiv · show

We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own self-critiques as feedback, refining their own summaries into better ones. Finally, we motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability. Our measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results are a proof of concept for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. We release our training datasets, as well as samples from our critique assistance experiments.

1 Introduction

The paper addresses scalable oversight by training models to critique outputs and assist humans evaluating difficult tasks. It studies whether critique assistance, scaling, self-critiquing, refinement, and critique-performance gaps can improve supervision.

  • Motivation: Human evaluation of code correctness and factual veracity requires substantial expertise, creating a scalable-oversight problem for difficult tasks.
  • Critique assistance: AI-assisted critiques help labelers find flaws in model- and human-written answers, including deliberately misleading summaries.
  • Scaling critique quality: Larger models generally produce more helpful critiques and better self-critiques despite having harder-to-critique answers.
  • Critique-conditional refinement: Larger models can use generated critiques to improve their own answers, outperforming direct refinement baselines as improvement scales with model size.
  • Measuring critique gaps: The paper proposes generator-discriminator-critique gaps and finds no clear evidence that critique performance catches up to discrimination performance.
  • Resources: The authors release training data and assistance-experiment samples, including misleading answers and intended flaws.

2 Dataset collection and model training

The authors jointly train language models on base, critique, discrimination, and refinement tasks using human demonstrations and synthetic tasks with oracle validity judgments. They define model scale by fine-tuning loss rather than parameter count.

  • Task structure: The training framework jointly models a base task, critique generation, answer critiqueability, critique helpfulness, and conditional refinement.
  • Topic-based summarization: Topic-based summarization asks for a summary focused on a specific aspect of a passage rather than the whole text.
  • Dataset collection: The dataset contains over 6,000 topical queries and summaries across more than 2,000 passages from short stories, Wikipedia, and web articles.
  • Demonstration collection: Demonstrations are collected in rounds, with models trained jointly on accumulated task data before generating inputs for later critique and helpfulness demonstrations.
  • Model scale: Model scale is defined as fine-tuning loss in nats per token, enabling comparison across architectures when parameter counts are not directly comparable.
  • Synthetic tasks: Synthetic tasks use binary ground truth and hand-coded oracles for base, critiqueability, critique, and helpfulness evaluation.
  • Synthetic-task training: Synthetic models are trained first on 100,000 generated base tasks, then on 100,000 critiqueability demonstrations and 50,000 critique demonstrations.

3 Assisting critique finding

The study evaluates natural-language critique assistance for human labelers reviewing topic-based summaries. Model-generated critiques increase the number and range of flaws labelers find, without appreciably increasing task time, though validity can be difficult to assess.

  • Experimental setup: Model assistance helps labelers find more critiques across model, human, and deliberately misleading answers.The experiment uses six conditions spanning three answer types and assisted versus unassisted critique finding.
  • Assistance procedure: Most assisted critiques reuse model suggestions, while only 15.3% of used critiques receive further editing.Critiques are de-duplicated, ranked by helpfulness, and presented with scores before labelers choose whether to use them.
  • Critique coverage: Assistance increases critiques across severity levels and categories, with the largest gains for minor and coverage critiques but additional major, critical, and accuracy critiques.The result indicates broader detection rather than improvement limited to one error type.
  • Perceived answer quality: With assistance, model answers receive critiques on 78% rather than 54% of inputs and average ratings fall from 5.6 to 4.9.Assisted labelers take slightly longer mainly because they write more critiques, while overall task time is not appreciably affected.
  • Limitations: A key confounder is that labelers may sometimes accept misleading critiques, because the experiment cannot always distinguish flawed answers from invalid critiques of good answers.The authors address this symmetry for deliberately misleading answers using intended critiques and independent validity judgments.
  • Validation checks: Assistance raises discovery of intended critiques from 27 ± 5% to 45 ± 5%, while author-judged validity remains 71% versus 73%.The number of valid critiques consequently increases from 0.62 ± 0.07 to 0.96 ± 0.09, alongside more valid and novel critiques at 0.24±0.06 versus 0.18±0.05.

4 Critique quality results

The paper finds that critique quality generally improves with model scale, including for self-critiquing and critique-conditioned refinement. However, synthetic-task trends are less clear, and refinement gains are noisy and depend on how critiques are selected.

  • Scaling critique quality: Larger models produce critiques humans rate as more helpful, including when self-critiquing answers made harder to critique.On topic-based summarization, larger models are better at self-critiquing even though their outputs are harder for humans and models to critique.
  • Critique-conditioned refinement: Larger models improve their own answers through critique-conditional refinement, outperforming direct refinement when critiques are selected by helpfulness.The comparison tests refinements conditioned on an answer and critique against refinements conditioned only on the answer.
  • Human and model comparisons: Our critique model outperforms InstructGPT baselines in helpfulness but remains substantially below human-written critiques.The comparison includes zero-shot and few-shot InstructGPT-style baselines.
  • Limitations: The self-critiquing result may be mildly confounded because the supervised dataset contains more critiques of outputs from larger models.The authors believe this effect is minimal.
  • Synthetic tasks: On synthetic tasks, critique-scaling trends are generally similar in critiqueable cases but less clear overall, leaving no strong basis for treating positive critique scaling as fundamental.The authors also do not know whether the trend would persist after reinforcement learning of both answer and critique models.
  • Refinement findings: Conditional refinement improves over direct refinement mainly with helpfulness-selected critiques, while larger N strengthens conditional refinement and direct refinement remains a stronger compute-controlled baseline.Both refinement forms significantly outperform original outputs for larger models but have little to no effect for smaller models.

5 Generator-discriminator-critique (GDC) gaps

The paper compares generation, discrimination, and critique abilities through GDC gaps, finding consistently positive GD and GC gaps but task-specific CD gaps that do not clearly close with scale.

  • Definitions: G, D, and C measure answer generation, answer discrimination, and answer critiquing, respectively.The study asks whether models can identify poor outputs and articulate their flaws to humans.
  • Method: The proposed XY gap measures how much Y performance exceeds X performance on a common axis.GC measures self-critiquing effectiveness, GD measures recognizing poor answers, and CD compares critique with discrimination.
  • Method: Critiques judge answers by generating a critique and treating an answer as poor when a human validates any critique as valid and severe.The primary results use N = 2, with human Likert ratings or binary uncritiqueability measures for sample quality.
  • Results: Positive GD and GC gaps appear consistently except when models are completely unable to perform the task, while CD gaps vary by task.CD is positive for topic-based summarization and 3-SAT, but negative for Addition and RACE.
  • Results: For topic-based summarization, the CD gap is approximately constant across model scale.Across most synthetic tasks it may decrease with model size, whereas RACE shows the opposite pattern because critiquing is close to oracle performance.
  • Implications: Overall, the gaps are task-specific, and the experiments do not show a clear general trend of critique performance catching up to discriminator performance.The authors suggest that difficult realistic tasks may make the CD gap harder to close, partly because discriminators can exploit labeler writing style in summarization.

6 Related work

Related work covers scalable alignment experiments, natural-language assistance for humans, critique datasets and models, and natural-language refinements across several domains.

  • Scalable alignment experiments: Prior scalable alignment work includes iterative amplification and debate experiments on algorithmic and realistic tasks, with mixed results in human-judged settings.Examples include sparse MNIST classification, movie-review claims, physics problems, and reading comprehension.
  • Human assistance with natural language: Natural-language assistance has been used to create challenging NLI datasets, find adversarial examples, and support passage-based question answering.These studies use model assistance for demonstrations, language-model classifications and generations, or human question-answering.
  • Human assistance with natural language: Other work helps humans evaluate text by accelerating fact-checking or discriminating whether text was model-generated.These approaches use natural-language briefs or language models to assist evaluation.
  • Critique datasets and models: Critique datasets and models include factual-claim evidence, critical peer reviews, mined disagreements, story critiques, and model-generated debate rebuttals.Project Debater focuses on debating against humans rather than models, unlike the work discussed here.
  • Natural language refinements: Natural-language feedback has improved models in computer vision, program synthesis, summarization, security repair, and mathematics.The cited work includes human feedback and language-model-generated refinements across these domains.

7 Discussion

The authors present AI-assisted critique as a proof of concept for scalable oversight, while emphasizing important limits in evaluation, assumptions, task difficulty, and experimental setup.

  • Topic-based summarization showed significant gains from AI critique assistance despite not being difficult for human labelers to evaluate.
  • Large language models are capable of meaningfully assisting human evaluation, and larger models may improve at critiquing their own outputs.
  • Generator-discriminator-critique gaps offer a way to study alignment properties and the quality of base-task training signals.
  • Learning from natural-language feedback is feasible because models can condition on critiques to improve their own outputs.
  • Limitations: The study lacks robust objective ground truth for validating topic-based summaries and critiques, while labelers may make mistakes or trust summaries too much.
  • Limitations: Critique-based assistance assumes problematic outputs have articulable flaws and may not fit inherently subjective tasks with unreconcilable preferences.
  • Limitations: Positive results may not generalize to harder tasks because the base task was relatively easy for humans to evaluate.
  • The authors release training datasets and assistance-experiment samples, including misleading answers and intended flaws.

A.4 Auxiliary tasks

The auxiliary-task pipeline expands supervised training beyond critiques and refinements, using standardized task formatting and data augmentation while leaving some design choices for future work.

  • Auxiliary tasks: The final supervised training data includes question creation and corroboration tasks alongside the paper’s primary tasks.
  • Data augmentation: Training converts conditional refinement to direct refinement and corroboration quotes to answer quotes 50% of the time.
  • Task design: The authors replaced critique-slate generation with single-critique generation to simplify the setup.
  • Formatting: Examples use a fixed sequence of passage, question, answer, critiqueability, critique, helpfulness, and refinement.
  • Formatting: The formatting and task ordering may be task-dependent, and reordering made little difference in the synthetic RACE domain.
  • Proof-systems perspective: The section situates critiques as a natural assistance form from the perspective of proof systems.

B.2 Proof systems in practice

The proof-systems perspective relates human feedback schemes to levels of the polynomial hierarchy and interprets critique-like assistance as a route to evaluating increasingly complex claims.

  • Reinforcement learning from human preferences asks models to find solutions while humans verify them, corresponding empirically to NP/co-NP.
  • Two-step recursive reward modeling and debate correspond to the second level of the polynomial hierarchy, with n-step versions corresponding to its nth level.
  • Critique-like assistance can move verification upward by asking humans to recognize critiques rather than directly produce answers.
  • At the next level, an assistant could critique critiques while humans evaluate those higher-order critiques.
  • Limitations: The proof-systems analogy suggests assistance becomes less useful as models become more powerful, and iterative refinement is not clearly outside P.
  • Gaps: The framework compares generation, discrimination, and critiquing abilities through corresponding gaps and binary discrimination tasks.

C.2 Relevance to model training and scalable oversight

The paper frames critique ability as a training-relevant property for scalable oversight and proposes comparing generation, discrimination, and critique performance. These comparisons expose when models can produce good outputs without articulating why alternatives are poor.

  • Motivation: A negative GC gap can cause training to favor outputs that look good to humans but are actually worse.The proposed remedy is to train critique models toward C ≥ G.
  • Motivation: Closing the CD gap, C = D, would mean critiques explain to humans why poor outputs are poor.The paper presents this as stronger than merely explaining everything the model knows.
  • Implications: GDC gaps can be studied on synthetic tasks with known ground truth even when difficult real-world tasks lack reliable labels.The paper presents gap analysis as useful for assessing training signals and alignment properties.
  • Measurement: The GDC framework measures average generator performance, best-of-N performance against a discriminator, and best-of-N performance against critique severity.The framework compares generation, discrimination, and critique ability using these three quantities.
  • Measurement: C can exceed D when it uses a human oracle to check critique validity, making the comparison asymmetric.The authors therefore propose Cm, which uses a helpfulness model to judge critique quality instead.
  • Results: Negative CD gaps appeared on the Addition, Alphabetize, and RACE synthetic tasks, which the authors suspect results from C using the oracle.They did not investigate whether Cm would instead fall below D.

C.3.2 Alternative C definitions

Alternative definitions of critique performance vary how many critiques are generated and how much humans participate in judging them. Human review of multiple critiques performs best among the tested variants, but remains below discriminator ability.

  • Definitions: Ch8 shows humans eight critiques, while C8 shows humans one helpfulness-optimized critique and Cm8 removes humans from judging.These variants trade off multiple assistance pieces, critique optimization, and fully model-based scoring.
  • Gap interpretation: The GC gap tests robustness of using critiques as a training signal, while the GD gap tests robustness of the critiqueability score.Both gaps evaluate best-of-N behavior relative to best-of-1.
  • Human-in-the-loop variants: C8 > C, and Ch8 > C8, but both variants still fall short of discriminator ability.Best-of-8 improves the critique shown to humans, while showing eight critiques performs better than showing one optimized critique.
  • Model-only variants: Cm is competitive with C, whereas using more critiques does not clearly improve model-only scoring because Cm8 is not clearly better than Cm.The model-only results are described as noisy and potentially confusing.
  • Scaling: Best-of-N gains do not appear to improve with model size for GC, and best-of-8 may perform worse than best-of-4.The best-of-N comparison is noisy, although best-of-4 appears consistently better than best-of-2.

D 2-step debate

The paper evaluates a two-step debate-like process in which a generated critique determines whether an answer is accepted. On misleading-versus-honest summaries, helpfulness-based best-of-N scoring improves with N but remains below critiqueability scoring.

  • Setup: Two-step debate generates an answer and critique, then accepts the answer only when the critique is not helpful.The paper compares this judgment with humans judging answers directly.
  • Evaluation: The experiment uses paired misleading and honest answers, measuring accuracy in selecting the honest answer without relying on humans finding critiques.The largest model generated critiques at temperature 0.5.
  • Results: Best-of-N helpfulness scoring discriminates better as N increases but never matches critiqueability scoring.Its performance appears to asymptote below critiqueability as N approaches infinity.
  • Interpretation: These results suggest naive two-step debate performs worse than RLHP on this task.The authors describe this as a negative result that does not rule out debate with larger models.
  • Caveats: A helpfulness model may prioritize nit-picky rather than severe critiques, motivating a severity model or critique-conditional preference model.The paper also targets domains where critiqueability labels are especially difficult to provide.
  • Caveats: Debate may be harder to implement initially than recursive reward modeling because it requires a robust helpfulness model.Recursive reward modeling can retain humans interacting with multiple critiques even without robust critiques.
  • Comparison assistance: Model-written critiques produced only small, noisy improvements in comparison accuracy, unlike human-written critiques, which improved accuracy significantly.The comparison experiment required substantially more effort and labeling and was discontinued.
  • Quote ablation: Quotes did not increase critiques found and saved slightly under a minute, nearly matching longest-common-substring highlighting.The highlighting baseline achieved almost the same time savings.

E.3 Ablation of number of critiques

The ablation varies the number of model-generated critiques provided as assistance. Eight critiques yielded more critiques found than four, though the speed result for four was potentially better than unassisted review.

  • Ablation: With four critiques, finding critiques was possibly faster than in the unassisted setting.The result is qualified as uncertain in the paper.
  • Ablation: The eight-critiques condition found more flaws than the four-critiques condition.The effect on the number of critiques found was a little less than twice as large.
  • Data: The samples and assistance conditions were selected under stated random-sampling and inclusion constraints.The supplementary materials provide samples from these experiments and release the full assistance dataset.

F.3 Refinement samples

This section presents randomly selected samples from helpfulness, assistance, and refinement experiments, including model critiques, human ratings, and refinement examples. The samples illustrate critiques identifying factual errors, omissions, and irrelevant content in answers.

  • Sample organization: The appendix provides randomly selected samples from helpfulness, assistance, and refinement experiments, with additional samples referenced in Tables 26–35.The refinement examples include conditional refinements using random or best-of-8 critiques and direct refinements.
  • Refinement procedure: Refinement experiments compare refinements against original answers using critiqueability scores, while filtering for cases where all three refinements rank higher by a human labeler.The procedure avoids forcing refinement when the original answer may already be perfect.
  • Helpfulness samples: The domestic-violence example shows critiques flagging a missing answer and omitted statistics about violence and hostage situations.The cited critiques identify the need to include one-in-three lifetime prevalence and 80% of U.S. hostage situations involving domestic violence.
  • Helpfulness samples: Human and assisted critiques of the Mike answer differ in focus, identifying inaccuracies, coverage omissions, and extraneous information.Human critiques note the incorrect hiding claim and missing death; assisted critiques additionally identify the golf-course setting and irrelevant content.
  • Helpfulness samples: The Mike example shows critiques identifying an invented hiding incident, omission of death, and an incorrect description of the setting.Model helpfulness scores for these critiques are 0.71, 0.51, and 0.42 respectively.
  • Helpfulness samples: The blonde-woman example shows critiques correcting an underdescribed relationship and omissions concerning her appearance, history with William, and reaction to his crimes.Assisted critiques also flag irrelevant appearance details and inaccuracies in the account of the elevator interaction.
Loading 2206.05802v2…