Source-linked AI summary
Debate Helps Weak-to-Strong Generalization
Hao Lang, Fei Huang, Yongbin Li
TL;DR
Human supervision may become inadequate as models exceed human capability. The paper combines debate-based scalable oversight with weak-to-strong generalization by improving a weak supervisor with strong-model information before training the strong model. The approach improves alignment across OpenAI weak-to-strong NLP benchmarks, but the experiments use relatively small model gaps and an expensive debate procedure.
Problem
Human evaluation may become unreliable for increasingly capable models, limiting existing alignment approaches and weakening their training signals.
Method
The paper uses debate between strong models and ensembles of weak models to improve weak supervision before fine-tuning a strong model on weak-generated labels.
Results
The approach outperforms strong-student baselines on all four tasks and raises PGR from 41.2% to 76.5% in SciQ, 56.4% to 69.2% in BoolQ, 17.4% to 56.5% in CosmosQA, and 35.0% to 70.0% in AnthropicHH.
Takeaways & Limitations
Debate has potential to improve weak-to-strong generalization by helping weak models extract trustworthy information from strong models.
Takeaways & Limitations
The experiments use only size-based weak–strong differences with relatively small compute gaps, and the approach requires two debaters plus multi-turn debate.
Abstract
from arXiv · showhide
Common methods for aligning already-capable models with desired behavior rely on the ability of humans to provide supervision. However, future superhuman models will surpass the capability of humans. Therefore, humans will only be able to weakly supervise superhuman models. This expected deficiency of human evaluation would weaken the safety of future AI systems. Scalable oversight and weak-to-strong generalization are two complementary approaches to tackle this issue. In this paper, we attempt to combine the strengths of these two approaches to further improve alignment. Specifically, we investigate ways of improving human supervision with a strong pretrained model and then supervise the strong model with enhanced weak human supervision. To make iterative empirical progress, we consider an analogy: can we use a strong model to improve weak model supervision and then use it to supervise the strong model? We empirically test it by finetuning a small weak model on ground truth labels with the additional help from a large strong model, and then finetuning the strong model on labels generated by the weak model. We find that debate can assist a weak model in extracting trustworthy information from an untrustworthy strong model, which provides leverage as context on samples when training a weak model. We also show that an ensemble of weak models helps exploit long arguments generated by strong model debaters and obtain a more robust supervision estimate. Extensive experiments on the OpenAI weak-to-strong NLP benchmarks show that the combination approach leads to better alignment, which indicates that debate has the potential to help weak-to-strong generalization.
Introduction
The paper combines scalable oversight with weak-to-strong generalization to address human supervision becoming unreliable for increasingly capable models. It uses debate and weak-model ensembles to improve supervision, reporting better alignment on OpenAI weak-to-strong NLP benchmarks.
- Motivation: Human evaluation may become unreliable as models surpass human capability, limiting existing alignment methods and potentially weakening their training signals.
- Approach: The paper combines scalable oversight, which improves human supervision, with weak-to-strong generalization, which trains strong models from weak supervision.
- Approach: The study tests a pipeline that uses a strong model to enhance a weak supervisor, then trains the strong model on labels generated by that weak supervisor.
- Findings: Debate helps weak models extract trustworthy information from capable but untrustworthy strong models, while ensembles improve robustness to long debate arguments.
- Findings: The combination improves alignment on OpenAI weak-to-strong NLP benchmarks, while the paper identifies hybrid superhuman alignment as an area for further research.
Related Work
The related work spans AI alignment, scalable oversight, weak-to-strong generalization, debate, and ensemble methods. This paper differs by using debate to train a better weak supervisor for subsequent strong-model alignment.
- AI alignment: AI alignment methods steer capable models toward human values using human demonstrations, reinforcement learning from human feedback, or direct alignment techniques.
- Scalable oversight: Scalable oversight improves human supervision of more capable models by exploiting structures such as easier evaluation than generation or task decomposability.
- Debate: Prior debate studies target inference-time judge accuracy, whereas this work uses debate to train a weak supervisor that can help align strong models.
- Weak-to-strong generalization: Weak-to-strong generalization fine-tunes strong pretrained models from weak supervision, relying on their representations of alignment-relevant tasks.
- Ensemble methods: Unlike prior ensemble work, this paper uses weak models with different seeds to exploit long debate arguments and improve supervision robustness.
Preliminaries
The weak-to-strong pipeline creates a weak supervisor, trains a strong student from its labels, and measures recovery relative to a strong ceiling trained on ground truth.
- Pipeline: The weak supervisor is a small pretrained model fine-tuned on ground-truth labels, with its performance called weak performance.
- Pipeline: The strong student is a large pretrained model fine-tuned on labels generated by the weak supervisor, with its performance called weak-to-strong performance.
- Pipeline: The strong ceiling is a large pretrained model fine-tuned on ground-truth labels, and its resulting performance defines the strong ceiling performance.
- Metric: Performance gap recovered measures the fraction of the gap between weak and strong-ceiling performance recovered by the strong student.
Methods
The method generates multi-turn debates between strong-model debaters, appends transcripts to weak-model training samples, aggregates weak-model ensembles, and uses their labels to train a strong student.
- Overview: The pipeline generates debate arguments, trains a weak-model ensemble on debate-augmented samples, and trains a strong student on ensemble-constructed labels.
- Debate protocol: Two large-model debaters receive opposing answers and exchange arguments over a fixed number of turns; this work uses three turns.
- Debate protocol: Debate is intended to expose false claims because an opposing debater can identify flaws, making lies harder to sustain than to refute.
- Weak supervision: Each weak-model sample is augmented with the debate transcript, but long transcripts may exceed a weak model’s processing capacity.
- Ensembling: Debate ensembles use different debate-sampling seeds, whereas finetune ensembles share transcripts but vary fine-tuning seeds; debate ensembles are costlier but more diverse.
- Strong student: The strong student is fine-tuned on weak labels formed by averaging predictions from the weak models in the ensemble.
Experiments
Across four binary NLP benchmarks, debate-based supervision improves weak-to-strong generalization, with debate ensembles outperforming strong-student baselines and alternative oversight variants. Ablations indicate that ensemble diversity helps exploit debate transcripts, while longer debates can hurt performance.
- Main Results: 76.5% PGR in SciQ, 69.2% in BoolQ, 56.5% in CosmosQA, and 70.0% in AnthropicHH improve on Finetune w/ aux. loss baselines.The corresponding baseline PGRs are 41.2%, 56.4%, 17.4%, and 35.0%.
- Main Results: Debate ensembles significantly outperform every strong-student baseline across all four tasks and achieve the best test accuracy among compared strong students.The baselines include naive weak-label finetuning and variants with confidence loss terms.
- Ablation on scalable oversight: Debate outperforms consultancy and market-making in test accuracy and PGR across all four tasks, supporting debate as the strongest tested oversight variant.Consultancy is a relatively weak comparison because it randomly assigns the consultant one answer, including the incorrect answer half the time.
- Ablation on weak model ensemble: Debate ensembles consistently outperform single weak models and finetune ensembles across tasks, whereas finetune ensembles improve over single models in three of four tasks.The results suggest that diversity in generated debate arguments improves the robustness of weak supervision, while a single weak model may not fully exploit long arguments.
- Ablation on ensemble cardinality: Four-member ensembles appear sufficient: performance shows a significant gap from three to four members but remains similar for four-, five-, and six-member ensembles.This cardinality ablation was conducted on SciQ and AnthropicHH.
- Ablation on debate turns: Three debate turns are used because they provide the minimum interaction for mutual critique, while accuracy consistently decreases after three turns as transcripts lengthen.The strong debaters struggle to process long debate transcripts and follow instructions in later turns.
Limitations and Conclusion
The paper presents debate as an effective combination of scalable oversight and weak-to-strong generalization, while emphasizing that this is only one possible hybrid approach. Its conclusions are limited by the modest model gap and computationally expensive multi-turn debate setup.
- Limitations: The proposed combination is only one of many ways to integrate scalable oversight and weak-to-strong generalization.The paper specifically mentions task decomposition plus W2SG as another possibility.
- Limitations: The experiments compare models differing only in pretrained-model size, not reasoning or planning abilities.The weak and strong models are 7B and 14B, respectively.
- Limitations: The approach is computationally expensive because it requires two debater instances and a multi-turn debate procedure.
- Conclusion: The approach improves weak-to-strong generalization by combining scalable oversight with weak-to-strong methods.