Source-linked AI summary
LLM Critics Help Catch LLM Bugs
Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, Jan Leike
TL;DR
Human evaluation is a fundamental limitation of RLHF as models become harder to assess reliably. This work trains RLHF-based LLM critics to help humans evaluate model-written code, finding that critics catch more bugs and that human-machine teams reduce hallucinations relative to models alone. The approach is promising but remains limited by its evaluation setting and substantial residual hallucination rates.
Problem
RLHF depends on humans correctly evaluating model outputs, but increasingly capable models may exceed reliable human evaluation ability.
Method
The authors train LLM critics with RLHF to generate natural-language critiques identifying errors in answers to real-world assistant coding tasks.
Results
Critics catch more inserted bugs than human contractors, while human-machine teams produce more comprehensive critiques and reduce hallucinations compared with models alone.
Takeaways & Limitations
LLM critics are a promising scalable-oversight approach for helping humans evaluate model outputs as model capabilities increase.
Takeaways & Limitations
Evaluations use typically short code snippets without multi-file support or repository navigation, and hallucinated bugs remain frequent despite reduction.
Abstract
from arXiv · showhide
Reinforcement learning from human feedback (RLHF) is fundamentally limited by the capacity of humans to correctly evaluate model output. To improve human evaluation ability and overcome that limitation this work trains "critic" models that help humans to more accurately evaluate model-written code. These critics are themselves LLMs trained with RLHF to write natural language feedback highlighting problems in code from real-world assistant tasks. On code containing naturally occurring LLM errors model-written critiques are preferred over human critiques in 63% of cases, and human evaluation finds that models catch more bugs than human contractors paid for code review. We further confirm that our fine-tuned LLM critics can successfully identify hundreds of errors in ChatGPT training data rated as "flawless", even though the majority of those tasks are non-code tasks and thus out-of-distribution for the critic model. Critics can have limitations of their own, including hallucinated bugs that could mislead humans into making mistakes they might have otherwise avoided, but human-machine teams of critics and contractors catch similar numbers of bugs to LLM critics while hallucinating less than LLMs alone.
1 Introduction
As model capabilities outpace reliable human evaluation, this work applies scalable oversight to real-world code tasks by training LLM critics to identify errors. CriticGPT and human-machine teams improve bug detection and critique quality relative to human or model-only baselines.
- Motivation: Human evaluation may become unreliable as models grow more capable, motivating scalable oversight methods that help humans assess model outputs.Earlier oversight work mainly targeted toy reading-comprehension settings, whereas this study focuses on economically important real-world code tasks.
- Results: LLM critiques catch substantially more inserted bugs than qualified human contractors paid for code review.The comparison uses code containing human-inserted bugs and contractors representative of those used in production.
- Approach: CriticGPT takes a question-answer pair as input and generates natural-language critiques that identify specific errors in the answer.The model is trained with RLHF on challenging real-world data.
- Results: Model-written critiques are preferred over human critiques more than 80% of the time on code with human-inserted bugs.The study summarizes pairwise critique preferences using model comparisons and Elo scores.
- Results: Human-machine teams write more comprehensive critiques while reducing hallucinations relative to models alone.The teams move beyond the model-only frontier by better avoiding nitpicks and hallucinations.
- Approach: Force Sampling Beam Search balances the number of real and spurious issues included in generated critiques.The inference-time procedure produces longer, more comprehensive critiques with a reduced rate of hallucinations or nitpicks.
2 Methods
The paper trains LLM critics through RLHF to identify problems in model-written code, evaluates them with structured critique attributes, and studies contractor assistance. Its data combine naturally occurring and contractor-inserted bugs, with inference-time Force Sampling Beam Search used to balance real issues against spurious ones.
- Critic models: Critic models accept question–answer pairs and produce structured critiques that quote answer spans and attach comments describing potential problems.Critiques can contain multiple quoted sections, each paired with comments about an error.
- Critique attributes: Contractors rate comprehensiveness, critique-bug inclusion, hallucinated bugs or nitpicks, and overall helpfulness on a 1–7 ordinal scale.The criteria capture both whether critiques find important issues and whether they introduce unsupported or minor problems.
- Critique attributes: Longer critiques tend to improve bug coverage but also increase hallucinations and nitpicks, creating a tradeoff between comprehensiveness and accuracy.Contractors were instructed to favor critiques that are accurate, comprehensive, concise, and avoid nitpicks.
- Critique comparisons: Pairwise critique comparisons are summarized with Elo scores, whose fitted win-rate model estimates how often one model’s critique is preferred over another’s.Ties count as half a win and half a loss, with confidence intervals obtained by nonparametric bootstrap.
- Human-machine critique teams: Human-machine studies pre-seed contractors’ responses with LLM critiques, allowing them to retain, remove, edit, or supplement suggested comments.Keeping comments unchanged and removing them were common, while adding comments and editing phrasing were less common.
- Evaluation data: The evaluation uses code from the RLHF pipeline, including Human Inserted Bugs created by contractor tampering and Human Detected Bugs drawn from previously identified coding errors.Inserted bugs are harder for humans to catch and more severe on average, whereas detected bugs are more natural but typically easier for humans to spot.
- Training critics: Critics are trained with an RLHF pipeline that samples critiques, collects contractor ratings, trains a reward model, optimizes a policy with PPO, and applies FSBS at inference time.FSBS combines reward-model scoring with constrained sampling to generate longer, more comprehensive critiques while reducing hallucinations or nitpicks.
3 Results
CriticGPT improves critique quality and bug detection over representative human contractors, while human-machine teams improve comprehensiveness and reduce hallucinations relative to models alone. FSBS exposes a comprehensiveness–spurious-claim tradeoff, and training-data and tampering choices materially affect performance.
- 3.1 LLM critiques are often preferred over human critiques and catch more inserted bugs: CriticGPT critiques were substantially preferred over prompted ChatGPT and contractor-written critiques, while both LLM critics caught more inserted bugs than human annotators.The comparison used representative contractors rather than an estimate of best-possible human performance.
- 3.1 LLM critiques are often preferred over human critiques and catch more inserted bugs: Contractors made fewer comments without LLM help despite a median handling time of 50 minutes, partly because some tasks required domain knowledge.The paper gives unfamiliar Python libraries as an example of missing domain knowledge.
- 3.2 CriticGPT RL training improves models across pre-training scale: Around 30x more pre-training compute was estimated to match CriticGPT’s RL-only performance using larger models without additional data or pipeline changes.The estimate comes from scaling a GPT-4-family model family trained with RLHF.
- 3.3 CriticGPT helps humans write more comprehensive critiques: Human+CriticGPT teams wrote more comprehensive critiques than humans alone, while CriticGPT improved comprehensiveness over ChatGPT on human-detected and inserted bugs.Models still had higher nitpick and hallucinated-bug rates than humans, although CriticGPT reduced those rates relative to ChatGPT.
- 3.4 FSBS lets us navigate tradeoffs between comprehensiveness and hallucinations: The probability of catching a bug increased with the number of critique claims, but more claims also increased the likelihood of nitpicks and hallucinations.The paper frames this as a precision–recall-like tradeoff for critique quality.
- 3.4 FSBS lets us navigate tradeoffs between comprehensiveness and hallucinations: FSBS uses reward models and test-time search to establish a Pareto curve between comprehensiveness and spurious claims without retraining the critique model.Human+CriticGPT teams moved beyond the model-only Pareto frontier in the Human Detected Bugs evaluation.
- 3.5 Ablations: CriticGPT trained on the paper’s data had higher precision and recall than a more closely matched ChatGPT reference on Human Detected Bugs.The closer comparison was designed to isolate data-collection effects from training duration and pipeline setup.
- 3.5 Ablations: Excluding tampered examples caused severe underperformance on both inserted and detected bugs in automated evaluations.The authors suspect lower agreement without reference bugs weakened the reward model.
4 Related Work
Related work spans self-correction, scalable oversight, code review, and directly trained critique models. This paper differs by applying RLHF with larger models to critique LLM-written code in challenging real-world settings.
- Self-correction: Self-correction methods use additional computation to improve final model responses, with clearer success when critique uses side-channel information.The cited work covers harmlessness, factuality, computer control, moral reasoning, and other domains.
- Scalable oversight: Scalable oversight instead aims to improve human judges’ ability to assess model answers, including through debate, recursive reward modeling, and market making.Prior empirical demonstrations include human-machine teams improving accuracy on benchmark tasks.
- Code review: Earlier deep-learning code-review systems primarily improved human-written code, whereas this system reviews code written by LLMs using more powerful pre-trained models.The distinction is the reviewed artifact and the model capability available to the system.
- Critique models: Compared with Saunders et al., CriticGPT uses RLHF, larger models, and a more challenging real-world evaluation setting.Saunders et al. is identified as the closest predecessor because it also trains models to produce critiques.
5 Discussion & Limitations
The evaluation assumes that helping contractors catch more bugs will improve downstream RLHF data, but the paper identifies important scope and measurement limitations. These include mismatched bug distributions, short single-file snippets, persistent hallucinations, and possible dual-use risks.
- Discussion: The evaluation assumes that helping contractors catch more bugs will produce improved policies, but the ideal comparison of critique-assisted versus unassisted RLHF is too expensive.Critics may also introduce consistent labeling biases that degrade RLHF performance.
- Limitations: Inserted bugs differ substantially from natural LLM errors, so results on tampered examples may not fully represent real-world error detection.The authors suggest training models to insert subtler in-distribution problems as future mitigation.
- Limitations: The evaluations use typically short code snippets without multi-file support or repository navigation, limiting representativeness for future coding agents.The paper also notes that complex bugs distributed across many lines were not investigated.
- Limitations: Although the method reduces nitpicks and hallucinated bugs, their absolute rate remains high, and a single critique step may be weaker than interactive procedures.The paper lists consultancy and debate as examples of multi-step alternatives.
- Risks: Strong bug-detection systems could be dual-use by helping attackers with source-code access find exploits they otherwise could not.The authors do not believe CriticGPT has improved bug detection enough to change the cybersecurity landscape.
6 Conclusion
LLM critics can help catch bugs in real-world model outputs, addressing the growing gap between model capability and human evaluation capacity. The authors present critics as a promising approach for scalable oversight.
- Scalable oversight is increasingly needed because humans may be unable to reliably evaluate more capable models.
- LLM critics catch bugs in real-world data and can assist human annotators.
7 Appendix
The appendix details evaluation procedures, critique-generation methods, and analyses of model and contractor agreement. It also compares model performance across bug distributions and critique-quality objectives.
- Force Sampling Beam Search (FSBS) Details: FSBS improves critique performance by sampling continuations, selecting high-scoring completions, and continuing generation from the selected outputs.
- Force Sampling Beam Search (FSBS) Details: Varying the length modifier trades critique comprehensiveness against hallucinations and nitpicks through reward-model reranking.
- Human Detected Bugs are easier to catch and less severe than Human Inserted Bugs.
- Contractors agree more on whether critiques include reference bugs than on other critique attributes or overall quality preferences.
- Figure 12 compares CBI performance on Human Inserted Bugs with comprehensive-critique rates on Human Detected Bugs.
7.6 Generalization of critique-bug inclusion (CBI) Metric
The appendix examines whether critique-bug inclusion generalizes across bug distributions and whether discriminator performance explains critic performance. Results show distribution-sensitive behavior and weak alignment between discriminator confidence and critique success.
- Generalization of critique-bug inclusion (CBI) Metric: CriticGPT performs better relative to ChatGPT on Human Inserted Bugs than on Human Detected Bugs under the CBI comparison.
- Generalization of critique-bug inclusion (CBI) Metric: FSBS variants generally improve both the CBI and comprehensive-critique objectives against their respective baselines.
- Discriminator Critique Gap: Small models have difficulty detecting inserted bugs with discriminator training.
- Judge Accuracy and Ground Truth Reward Experiments: Current LLM critics did not significantly help humans identify correct solutions in challenging competition-code pairs.
- Discriminator Critique Gap: On the 10% of cases where the largest discriminator was most confidently wrong, critics still caught the inserted bug around 54% of the time.
7.8 Lessons from Judge Accuracy and Ground Truth Reward Experiments
Ground-truth experiments tested whether critiques help humans judge paired passing and failing solutions on challenging code problems. In this setting, critics struggled to improve human identification of the correct solution.
- Experimental setup: Contractors judged paired passing and failing solutions without code execution, using sampled critiques and a 20-minute time limit.
- Results: LLM critics broadly struggled to help humans identify the correct solution in challenging competition-code tasks.
- Interpretation: The authors suspect an asymmetry between the compute used to find difficult solutions and the compute used to produce critiques.
- Why code: Code provides relatively crisp evaluation and practical importance because buggy or insecure generated code can compromise production systems.
7.11 Future Directions
The authors identify future directions around evaluation, deployment, data, and more realistic tampering. They also note that current measurements and studies leave important gaps.
- The work focused on production training data and therefore did not release its dataset.
- The authors highlight challenging tasks with low-noise expert labels as valuable for testing scalable oversight methods.
- Longitudinal production studies should track contractor productivity and final policy quality.
- Training models to tamper with code could produce subtle errors closer to the assistant’s usual output distribution.
- Human annotator performance was measured indirectly through critique-bug inclusion rate, with direct measurement identified as a future direction.
7.12 Contributions
The contributions describe the team’s work on RL experiments, critique evaluation, human data collection, tampering, critic training, and manuscript development.
- Nat McAleese ran early RL experiments, managed the team, provided research guidance, wrote much of the manuscript, and implemented interfaces for tampering and code review.
- Rai produced motivating results, designed critique comparisons, built key critique evaluations, studied tampered data and human accuracy, and explored generalization to general assistance.
- Juan Felipe Cerón Uribe led substantial human data collection, implemented adversarial tampering, proposed assisted-human evaluation methodology, and contributed to the manuscript.
- Evgenia Nitishinskaya trained the critic models presented apart from ChatGPT and ablated features of RL training.
7.13 CriticGPT Samples
The samples show CriticGPT identifying genuine security and correctness concerns in human- and model-written code, while sometimes overstating or hallucinating problems. Across examples, its critiques focus on file access, exception handling, path containment, and cryptographic use.
- CriticGPT correctly flags startswith()-based path containment as insecure because symlinks or similarly named directories can bypass the check.
- CriticGPT notes that opening a file only in read mode excludes write or other access use cases.
- CriticGPT identifies information exposure and unsafe broad exception handling in file-access code.
- The examples also document overstatement and hallucinated bugs, including a cryptographic critique identified as containing at least one hallucinated bug.
- CriticGPT warns that omitting an authentication tag when decrypting can prevent message integrity and authenticity verification.
- In a separate encryption example, CriticGPT comments on absent input validation and returning decrypted bytes without decoding them.