Source-linked AI summary
Measuring Progress on Scalable Oversight for Large Language Models
Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Liane Lovitt, Nelson Elhage, Nicholas Schiefer, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Robin Larson, Sam McCandlish, Sandipan Kundu, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Ben Mann, Jared Kaplan
TL;DR
Scalable oversight requires reliable supervision of models that may possess capabilities humans lack, but empirical study is difficult before such systems exist. This paper develops a sandwiching-based experimental design and tests a simple chat-assistance baseline on MMLU and time-limited QuALITY. Human–model teams substantially outperform both models alone and unaided humans, although the experiment’s relaxations limit how broadly the result can be applied.
Problem
Scalable oversight needs empirical methods for supervising models that may outperform humans on task-relevant skills, yet current systems do not broadly exceed human abilities.
Method
The paper operationalizes sandwiching experiments and tests human participants solving difficult MMLU and time-limited QuALITY questions with a dialog-model assistant.
Results
Human–model teams substantially outperform both models alone and unaided humans on MMLU and time-limited QuALITY.
Takeaways & Limitations
The findings provide a proof of concept that sandwiching-style scalable oversight research is viable with present models and that dialog models can assist humans with difficult tasks.
Takeaways & Limitations
The experiment does not fully implement sandwiching, using controlled multiple-choice tasks, no further fine-tuning, and answer reveals after each question.
Abstract
from arXiv · showhide
Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks.
1 Introduction
The paper frames scalable oversight as reliable supervision that remains effective when models possess important capabilities humans lack. It introduces sandwiching-style empirical studies and a proof-of-concept showing that human–model interaction can improve difficult question answering.
- Motivation: Scalable oversight seeks supervision that remains effective after models reach broadly human-level performance.The supervision may take the form of labels, reward signals, or critiques.
- Findings: Human participants using a simple conversational strategy produced high-quality answers on both tasks, reinforcing evidence that language models can assist humans with difficult work.The authors describe the strategy as an obvious starting point rather than the technique they ultimately expect to be best.
- Paradigm: Sandwiching places model capability between typical humans, who struggle unaided, and specialists, who can reliably solve the task.Experts provide a reference for evaluating whether non-experts can successfully use or train the model.
- Motivation: The paper addresses a gap in empirical evidence for proposed scalable oversight techniques such as debate and market making.The authors present sandwiching as a way to gain evidence and practical experience with these approaches.
- Experiment: The proof-of-concept asks humans to solve difficult MMLU and time-limited QuALITY questions with a large-language-model assistant.The design tests whether present tasks satisfy the conditions needed for sandwiching-style research.
2 A Research Paradigm for Scalable Oversight
The sandwiching paradigm studies oversight by placing a model between expert evaluators and non-expert participants, then using expert review to assess whether participants successfully elicited reliable model behavior. The initial experiments simplify the paradigm by restricting interaction to dialogue and replacing test-time experts with benchmark labels, which narrows the conclusions.
- The Paradigm: Sandwiching selects tasks where models outperform typical humans but remain less capable than experts.This arrangement makes it possible to study oversight before systems broadly exceed human abilities.
- The Paradigm: Expert evaluators provide an upper bound on supervision quality, while non-experts use scalable techniques to build justified confidence in the model’s performance.The model is expected to possess relevant capabilities but not reliably apply them in aligned ways.
- Research Agenda: A full agenda uses an inner loop of non-expert alignment attempts and an outer loop in which experts review the resulting model.The long-term goal is to make the inner loop reliable without grounded feedback from experts.
- Relaxations: The initial experiments restrict participants to dialogue with a fixed assistant, excluding inspection and further fine-tuning.Success in this relaxed setting provides positive evidence, but rules out many potentially viable oversight strategies.
- Relaxations: Replacing experts with benchmark labels enables automatic evaluation but limits measurement of how close performance is to the best an expert could elicit.The authors treat incremental progress in this relaxed setting as progress toward the broader problem.
- Potential Techniques: The paradigm can support techniques ranging from plain model interaction to debate and market making.Plain interaction relies on eliciting knowledge, checking consistency, reviewing outputs, and synthesizing findings with human judgment.
3 A Simple Experiment
The experiment tests whether MMLU and time-limited QuALITY create useful sandwiching conditions with a current dialog model. It combines difficult specialized or long-passage questions with chat-based assistance, confidence reports, and controlled participant procedures.
- Tasks: MMLU tests specialized multiple-choice knowledge that many participants are unlikely to know, while the model may possess relevant domain knowledge from pretraining.The task is designed to leave room for both model knowledge and human synthesis.
- Tasks: QuALITY tests multiple-choice reading comprehension over roughly 5,000-word stories under a time limit.The questions are intended for college-educated adults but ordinarily require 15–30 minutes of thorough reading.
- Participants and Scale: The study uses two groups of five participants, collects 100 randomly sampled validation questions per experiment, and prioritizes work quality over quantity.Results later exclude one participant per task because their unaided accuracy was much higher than the other participants’ accuracy.
- Interface: The interface combines a model chat pane with a question, four answer choices, confidence ratings, and a reset function for reevaluating the model.For QuALITY, the passage is inserted into the conversation and a timer automatically submits the current answer when it expires.
- Methods: Participants receive the correct answer after each submission, a deliberate deviation that helps them learn which interaction strategies work.This feedback has no analog in settings where oversight must operate under genuine uncertainty.
- Experimental Conditions: Assisted performance is evaluated against unassisted humans and model baselines, with participants completing the assisted condition before the unassisted condition.That ordering means any learning effect would bias results toward a null effect rather than exaggerating assistance gains.
- Experimental Conditions: The study reports unassisted human, model, few-shot model, and model-assisted human conditions using a 52B-parameter RLHF-fine-tuned dialog assistant.The MMLU model condition includes five-shot context, whereas QuALITY cannot use five-shot examples because of input-length limits.
4 Results and Discussion
Human–model teams substantially outperform both unaided humans and the best model-only baselines on MMLU and timed QuALITY, while qualitative findings show that participants developed systematic ways to exploit and verify model outputs. The results also expose limits in model reasoning, calibration, and the experiment’s scope.
- Quantitative results: 75% vs. 66% on MMLU and 77% vs. 67% on QuALITY: model-assisted humans outperformed the best model-only results on both tasks.The best model results were 66% on MMLU and 67% on QuALITY.
- Quantitative results: 78% vs. 66% on MMLU and 86% vs. 67% on QuALITY: weighted-majority human–model teams exceeded the best model results on both tasks.On timed QuALITY, model-assisted teams reached 86%, below the 94% untimed human-team result reported by Pang et al.
- Model baselines: Best-of-20 chain-of-thought prompting improved model performance to 66% vs. 62% on MMLU and 67% vs. 59% on QuALITY.Few-shot learning also improved MMLU performance to 62% vs. 57%.
- Interaction patterns: Human–machine dialogs averaged 10.9 turns and 2.0 resets on MMLU versus 6.3 turns and 0.1 resets on QuALITY.The lower QuALITY figures were likely explained substantially by its time limit.
- Qualitative findings: Participants combined fact retrieval, answer-choice interrogation, explicit reasoning, resets, multiple strategies, and passage quotes to check model outputs.They found factual questions easier for the model than questions open to interpretation.
- Qualitative findings: The model updated assumptions after corrections, helping participants repair reasoning errors but also making it overly deferential to participant misunderstandings.Participants therefore had to distinguish useful correction from agreement that merely followed their own mistaken assumptions.
5 Related Work
The paper connects scalable oversight to human–AI team decision-making, distinguishing its interactive, general-purpose natural-language setting from prior targeted case studies. Related work shows that simple explanations can inappropriately increase trust and reduce team performance.
- Human–AI collaboration: This work relates to human–AI team decision-making research spanning recidivism prediction, medical diagnosis, and credit risk prediction.Its distinctive setting uses a more general-purpose AI system with a natural-language interface.
- Human–AI collaboration: Participants can interactively probe a model’s knowledge or reasoning for a specific task and question through natural language.
- Findings from related work: Simple non-interactive explanations can increase users’ trust in AI systems to an inappropriate degree, harming overall team performance.
- Findings from related work: Human–AI teams are less effective when the AI system’s capabilities and explanations create problematic collaboration conditions.
6 Conclusion
The conclusion argues that scalable oversight is necessary for safely deploying AI at or above human capability and presents this work as an empirical research paradigm toward that goal. Results from a simple chat protocol show that current language models can assist humans with difficult tasks, while leaving substantial room for improvement.
- 6 Conclusion: Safely deploying AI systems broadly at or above human capabilities will require progress on scalable oversight.
- 6 Conclusion: The paper introduces a research paradigm based on the sandwiching proposal and evaluates a simple human–model chat protocol for difficult questions.
- 6 Conclusion: Present-day language-model systems can assist humans with difficult tasks, although the reported results fall short of fully satisfactory performance.
- 6 Conclusion: The experiments establish that scalable oversight techniques can be productively evaluated on existing NLP task datasets.
- 6 Conclusion: The paper calls for empirical trials of debate, market-making, and recursive reward modeling to reduce risks and maximize the potential of general-purpose AI systems.
A Participant Instructions
Participants answer multiple-choice questions about long stories or articles with help from a capable but imperfect assistant under time limits. The instructions encourage flexible probing, verification, and resetting the assistant when it becomes anchored to an error.
- Task setup: For QuALITY and a variant of untimed MMLU, participants receive a long story or article and answer a multiple-choice question about it.
- Task setup: Participants should skim or spot-check the source text rather than read it completely, using it primarily for assistant support and verification.
- Time limit: The five-minute timer ends access to the conversation and submits the participant’s response.
- Assistant use: The assistant can be used flexibly for quotations, summaries, arguments, role-play, or critiques of previous responses.
- Assistant reset: Participants are advised to reset the assistant because mistakes can persist through later responses as conversational consistency.
B Random Example Dialogs
This section presents two randomly sampled dialogs for each task.
- B Random Example Dialogs: Two randomly sampled dialogs are included for each task.
B.1 MMLU
The MMLU passages show participants using dialog with the assistant to answer multiple-choice questions across legal and sociological topics. The assistant supplies definitions and applies them to candidate answers, though its responses are not uniformly reliable.
- Legal questions: The legal example asks whether a deed restriction on lot size and dwelling type is an easement, affirmative covenant, quiet-enjoyment covenant, or negative covenant.The restriction binds subsequent grantees and limits future subdivision and residential use.
- Legal questions: The assistant explains a negative covenant as a promise not to do certain things and applies it to restrictions on land use.It contrasts this with affirmative covenants, which require a positive action.
- Legal questions: The assistant also defines easements as limited rights to use another person’s land without transferring ownership.Its examples include utilities, roads, sewers, and crossing the property.
- Sociology questions: For Weber’s value-relevance question, the assistant attributes knowledge claims to researchers’ interests, biases, and values.The response says objective value-free knowledge is impossible because perspectives shape inquiry and interpretation.
B.2 QuALITY
The QuALITY passages illustrate time-pressured reading-comprehension interaction in which the assistant answers questions about an omitted story and related article excerpts. The dialogue includes factual claims, quotations, and occasional unsupported or incorrect responses.
- Story questions: The QuALITY setup asks readers to answer questions about long passages, including a question about Dennis’s consequences from spending a night in the Jovian Chamber.The displayed story material indicates distraction, emotional conflict, and pleasure-palace imagery, but the full passage is omitted.
- Story questions: The assistant claims that Koerber’s escape led to Dennis being grounded, losing his last chance at redemption, and later being marooned with his crew.These consequences are presented as a connected narrative explanation.
- Story questions: The assistant answers follow-up questions by describing Dennis’s relationship with Marla, the Jovian Chamber’s hypnotics, and his lack of rehabilitation.These responses connect the chamber to later career and personal consequences.
- Article questions: For the Inaugural Addresses passage, the assistant describes a shift from modest presidential addresses toward assertive leader-preacher rhetoric.It says later inaugurals increasingly inspire public behavior rather than merely describe government policy.
- Article questions: The assistant identifies a three-phase account of inaugural-address style and characterizes the third phase as still ongoing.The passage itself qualifies the classification while asserting that the overall trend is clear.
C Dialog Demonstrating Misleading Model Arguments
The dialog passages present questions about Lewin, Lippitt, and White’s 1939 leadership experiment and contain conflicting assistant descriptions of which leadership style produced the most work.
- Experiment description: One assistant response describes the experiment as comparing autocratic and democratic settings with preschool children using blocks.It emphasizes greater cooperation, initiative, and satisfaction in the democratic setting.
- Reported findings: The assistant states that laissez-faire leadership produced the least work and that democratic leadership was the most productive.It characterizes democratic leadership as combining autonomy with structure, guidance, and support.
- Reported findings: A later response instead says participative leadership produced the most work and describes authoritarian leadership as least effective.This answer conflicts with the preceding response’s terminology and ranking.