Source-linked AI summary
WebGPT: Browser-assisted question-answering with human feedback
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, John Schulman
TL;DR
Long-form question-answering systems must generate useful answers to open-ended questions, yet existing approaches lag behind human performance. WebGPT fine-tunes GPT-3 to browse the text-based web, collect references, and optimize answers using human demonstrations and preference feedback. Its best model is preferred to human demonstrators 56% of the time and to highest-voted ELI5 answers 69% of the time, while still struggling with out-of-distribution questions.
Problem
Long-form question-answering systems lag behind humans, motivating methods that address information retrieval and synthesis together.
Method
WebGPT fine-tunes GPT-3 to operate a text-based web browser, collect supporting references, imitate human demonstrations, and optimize answer quality using human feedback.
Results
56% of the time, the best model’s answers are preferred to human demonstrators’ answers, and 69% of the time they are preferred to the highest-voted ELI5 answers.
Takeaways & Limitations
WebGPT demonstrates that a fine-tuned language model can use browsing and references to outperform human demonstrators on ELI5 while improving over GPT-3 on TruthfulQA.
Takeaways & Limitations
WebGPT can appear more authoritative because of citations, creating potential overreliance, and it makes more mistakes than humans on out-of-distribution questions.
Abstract
from arXiv · showhide
We fine-tune GPT-3 to answer long-form questions using a text-based web-browsing environment, which allows the model to search and navigate the web. By setting up the task so that it can be performed by humans, we are able to train models on the task using imitation learning, and then optimize answer quality with human feedback. To make human evaluation of factual accuracy easier, models must collect references while browsing in support of their answers. We train and evaluate our models on ELI5, a dataset of questions asked by Reddit users. Our best model is obtained by fine-tuning GPT-3 using behavior cloning, and then performing rejection sampling against a reward model trained to predict human preferences. This model's answers are preferred by humans 56% of the time to those of our human demonstrators, and 69% of the time to the highest-voted answer from Reddit.
1 Introduction
WebGPT addresses long-form question-answering by combining web browsing, language-model synthesis, and human-feedback training. Its best model produces answers preferred to both human demonstrators and highly voted ELI5 answers, while also improving truthfulness over GPT-3 on TruthfulQA.
- Long-form question-answering requires paragraph-length responses to open-ended questions, but current systems lag behind humans and typically separate retrieval from synthesis.
- WebGPT combines a text-based browsing environment with imitation learning and reinforcement learning to improve retrieval and synthesis end to end.
- Human-collected demonstrations, preference comparisons, behavior cloning, reward modeling, reinforcement learning, and rejection sampling form the training pipeline.The best model combines behavior cloning with rejection sampling against the reward model.
- 56% of the time, the best model’s answers are preferred to human demonstrators’ answers, and 69% of the time they are preferred to the highest-voted ELI5 answers.
- On TruthfulQA, WebGPT answers are true 75% of the time and both true and informative 54% of the time, outperforming GPT-3 but falling short of humans.
2 Environment design
The environment uses a text-based interface in which the model operates a modern search engine through discrete browsing commands. It records selected web passages as references before the model composes its answer.
- WebGPT outsources document retrieval to Bing, allowing the model to focus on using a search engine to answer questions rather than improving retrieval itself.The approach relies on modern search engines’ large indexes and up-to-date documents.
- At each step, the model receives a summary containing the question, current page text, cursor location, and other environment information, then issues a command such as searching, clicking, or scrolling.
- The model can quote page extracts, which are stored with their title and domain as references used to compose the final answer.Browsing ends when the model stops, reaches the action limit, or reaches the reference-length limit, provided at least one reference exists.
3 Methods
WebGPT uses human demonstrations and preference comparisons to train GPT-3 models for browsing and answer generation. Training combines behavior cloning, reward modeling, reinforcement learning, and rejection sampling under disjoint data splits.
- 3.1 Data collection: Human demonstrations teach the model valid browser use, while preference comparisons provide direct supervision for answer quality.Comparisons ask humans to choose between two model-generated answers to the same question.
- 3.1 Data collection: Around 6,000 demonstrations and 21,500 comparisons were collected, with most questions drawn from ELI5.ELI5 supplied 92% of demonstration questions and 98% of comparison questions.
- Training methods: The study fine-tuned 760M, 13B, and 175B GPT-3 models because capabilities such as reading comprehension and synthesis emerge from pre-training.
- Training methods: Behavior cloning fine-tunes on demonstrations, while reward modeling predicts scalar preferences for answers with references.
4 Evaluation
WebGPT is evaluated on ELI5 against human demonstrations and Reddit’s highest-voted answers, and on TruthfulQA against base GPT-3 models. Its strongest ELI5 model is preferred over both comparison baselines, while all WebGPT models outperform GPT-3 on TruthfulQA.
- Evaluation setup: The evaluated WebGPT models use behavior cloning followed by same-size reward-model rejection sampling at 760M, 13B, and 175B scales.The models are best-of-4, best-of-16, and best-of-64 respectively, corresponding to different inference-time compute budgets.
- ELI5: ELI5 evaluations compare WebGPT with browser-use demonstrations and the highest-voted answer for each question, treating ties as 50% preference ratings.The comparisons use similar criteria for model and demonstrator answers, while reference-answer comparisons omit WebGPT references for fairness.
- TruthfulQA: All WebGPT models outperform all GPT-3 models on both the percentage of truthful answers and the percentage of truthful and informative answers in TruthfulQA.GPT-3 is evaluated with automated metrics, whereas WebGPT is evaluated by humans because its answers are out-of-distribution for the automated metric.
- TruthfulQA: The percentage of truthful and informative WebGPT answers increases with model size, unlike GPT-3 with either prompt.TruthfulQA answers are truncated to 50 tokens before evaluation, with trailing partial sentences removed.
5 Experiments
Experiments compare reinforcement learning and rejection sampling with behavior cloning, then examine how dataset size, model size, and sampling compute affect performance. Rejection sampling provides the largest reported benefit, while scaling results show gains from more data and compute.
- Comparison of training methods: 68% of the time, the 175B best-of-64 BC model is preferred to the 175B BC model, compared with 58% for the 175B RL model.Rejection sampling therefore provides a larger benefit than reinforcement learning in this comparison.
- Comparison of training methods: The combination of RL and rejection sampling offers little benefit over rejection sampling alone.The authors suggest shared reward-model optimization and reduced policy entropy as possible reasons.
- Scaling experiments: Doubling demonstrations increases policy reward-model score by about 0.13, while doubling comparisons increases reward-model accuracy by about 1.8%.These dataset-size trends are shown in the BC and RM scaling experiments.
- Scaling experiments: Doubling policy parameters increases reward-model score by roughly 0.09, while doubling reward-model parameters increases accuracy by roughly 0.4%.Parameter-count trends are noisier than dataset-size trends.
- Scaling experiments: Some rejection sampling is generally compute-efficient, but the most efficient trade-off uses limited sampling rather than the maximum amount.The reported Pareto-frontier models are 760M best-of-4, 13B best-of-16, and 175B best-of-64.
6 Discussion
WebGPT may reduce several kinds of false statements, but its authoritative citations, bias reinforcement, reference cherry-picking, and live web access create important risks and evaluation challenges.
- Truthfulness: WebGPT suggests fewer imitative falsehoods than GPT-3, yet it sometimes quotes highly unreliable sources on TruthfulQA.The authors hypothesize that the distribution shift from ELI5 to TruthfulQA contributes to this behavior.
- Truthfulness: WebGPT also suggests fewer non-imitative falsehoods than GPT-3, though paraphrasing and synthesis mistakes remain.The authors did not test this hypothesis directly because subtle hallucinations were difficult for labelers to identify.
- Perceived truthfulness: Citations can make WebGPT’s answers appear more authoritative, increasing the risk that users over-rely on mistakes, especially on out-of-distribution questions.The paper notes that WebGPT can make more mistakes than humans on such questions.
- Reinforcement of bias: WebGPT can reinforce biases inherited from GPT-3, existing sources, and the implicit assumptions or stance of questions.The authors connect question stance to potential confirmation-bias effects in users.
- Using references to evaluate factual accuracy: References improve feedback accuracy, reduce labeling noise, and increase transparency, but models may cherry-pick convincing sources rather than fairly assess evidence.The paper suggests debate-style methods that seek evidence both for and against claims as a possible mitigation.
- Evaluation: Evaluating factual accuracy for training remains difficult because practical criteria, including source trustworthiness, require contested judgment calls.The authors call for cross-disciplinary research to develop criteria that are both practical and epistemically sound.
- Risks of live web access: Live web access enables up-to-date answers but could expose users and others to risks from real-world side effects of model actions.The example given is editing Wikipedia to construct a reliable-looking reference if forms were accessible.
7 Related work
Related work combines language models with retrieved documents or treats browsing as reinforcement learning, while WebGPT targets long-form web question-answering.
- Earlier question-answering systems combined machine learning with external knowledge bases, including DeepQA, which was used to beat human Jeopardy performance.
- Dense passage retrieval trains retrieval contrastively, whereas REALM and RAG train retrieval and question answering end-to-end with language modeling objectives.
- Unlike these systems’ short-answer benchmarks, WebGPT addresses long-form question answering using web browsing.
- Other work formulates document retrieval or web browsing as reinforcement learning, including search and scrolling actions for question answering.
8 Conclusion
WebGPT fine-tunes a language model to browse the web for long-form question answering, uses collected references to support evaluation, and achieves strong ELI5 performance while struggling on out-of-distribution questions.
- WebGPT fine-tunes a language model to use a text-based web-browsing environment for long-form question answering.
- The approach directly optimizes answer quality with imitation learning and reinforcement learning, while requiring references collected during browsing.
- The best model outperforms humans on ELI5 but still struggles with out-of-distribution questions.
9 Author contributions
The listed contributors led the project, supported its code and infrastructure, analyzed bias, and provided security and networking assistance.
- Reiichiro Nakano, Jacob Hilton, Suchir Balaji, and John Schulman jointly led the project and wrote the paper.
- Jeff Wu, Long Ouyang, Xu Jiang, and Karl Cobbe provided advice across many topics.
- Jeff Wu, Vineet Kosaraju, William Saunders, and Xu Jiang made key contributions to the project codebase.
- Christina Kim, Christopher Hesse, and Shantanu Jain built and supported model-training and inference infrastructure.
- Tyna Eloundou and Gretchen Krueger analyzed bias and contributed to the paper.
- Kevin Button and Matthew Knight provided computer security support, while Benjamin Chess provided computer networking support.
- The browsing environment was written mostly in Python with some JavaScript and used Bing, Node.js, and Readability.js for web interaction.
B Question dataset details
The question data was drawn primarily from ELI5 and supplemented with smaller collections from TriviaQA, ARC, and handwritten questions. ELI5 questions were normalized through filtering, concatenation, and explicit question phrasing.
- Datasets: Most demonstration and comparison questions came from ELI5, with smaller additions from TriviaQA, ARC, and handwritten questions.ARC questions were converted from multiple-choice format into free-form prompts.
- ELI5 preprocessing: ELI5 preprocessing retained full URLs, removed deleted content, concatenated titles with selftext, and prepended “Explain: ” when needed.Questions were treated as explicit questions when they contained a question mark or selected question-word sequences.
- Dataset breakdown: The final numbers of demonstrations and comparisons for each dataset were reported in Table 4.The supplied passage identifies the table as the dataset-wise breakdown but does not provide its counts.
C Data collection details
Data collection combined contractors from Upwork and Surge AI with interfaces designed to approximate the model’s browsing conditions. Comparisons emphasized supported, useful, coherent answers while limiting training to the final overall preference rating.
- Contractors: Around 25% of data came from 10 Upwork contractors and around 75% from 46 Surge AI contractors, with the top five providing around 50%.Contractors were hired through Upwork and Surge AI to scale collection.
- Contractor process: Contractors received videos and detailed instructions, were generally highly educated, and were paid by hours worked rather than tasks completed.Demonstrations averaged around 15 minutes, while comparisons averaged around 10 minutes.
- Quality control: Quality control used paid trials, manual checks, researcher–labeler agreement monitoring, and labeler–labeler agreement monitoring.The final researcher–labeler agreement rate was 74%; the supplied passage truncates the labeler–labeler rate.
- Interfaces: The interfaces aimed to give humans the model’s information and actions, with exceptions for model action summaries and multi-step scrolling.Humans used a graphical interface, while comparison annotations included reference trustworthiness, claim support, relevance, and overall ratings.
- Comparison criteria: Comparisons judged unsupported information, question coverage, helpfulness, coherence, citation errors, and irrelevance, using a 5-point preference scale.Contractors judged support through reliable references or common knowledge rather than independent research.
- Training labels: Only the final comparison rating was used for training, with “much better” and “better” collapsed together.Auxiliary prediction of other annotations did not significantly improve reward-model validation accuracy.
- Contractor survey: Among surveyed contractors completing at least 32 tasks, most reported enjoyment, fair pay, and overall satisfaction, while a significant minority found the task repetitive.Three surveys produced 41 equally weighted ratings per question.
G TriviaQA evaluation
The paper evaluated WebGPT beyond its primary long-form setting and examined how question framing affects factuality, answer stance, and cultural assumptions. It also developed an estimator for predicting rejection-sampling preference at moderate sample sizes.
- TriviaQA evaluation: TriviaQA required additional fine-tuning to extract short-form answers conditioned on WebGPT output, using 256 questions because of test-train overlap concerns.An ablation fine-tuned GPT-3 without the WebGPT output.
- TriviaQA evaluation: WebGPT transferred to TriviaQA’s short-form setting, where it performed slightly better than UnitedQA-E without test-train overlap and slightly worse with overlap.The evaluation used the 175B behavior-cloning model, temperature 0.8, and no rejection sampling.
- Question stance: Affirming conspiracy or misconception beliefs tended to elicit inaccurate answers more often than neutral or skeptical questions, but the small experiment was not definitive.The study used 60 questions covering 10 conspiracy theories and 10 common misconceptions across three stances.
- Question stance: The question-stance experiment also examined whether answers explicitly affirmed or refuted the belief implied by each question.Answers came from three compute-efficient WebGPT models.
- Limitations: The authors note that the stance findings require further research because the experiment was small and may depend on framing, data collection, and stance intensity.They also encourage interdisciplinary study of other forms of bias observed in the wedding case study.
- Reference-point bias: For “What does a wedding look like?”, WebGPT often assumed a Western or American reference point: 20 of 64 answers mentioned America or American.Only four answers focused on a specific named non-American culture.
- Rejection-sampling prediction: The rejection-sampling estimator predicted human preference well for n ≤64, while sufficiently large n may be overestimated because the validation reward model becomes overoptimized.The method reuses samples across different n values by sorting them by the original reward-model score.
K Comparison dataset release details
The authors released 19,578 comparison records suitable for reward modeling. Each item contains paired records describing the shared question, browsing extracts, and composed answer.
- Release: The released dataset contains 19,578 comparisons marked suitable for reward modeling.The dataset is provided as a JSONL download.
- Record contents: Each comparison pairs two records with a shared question, browsing quotes and page titles, and a final answer composed from those quotes.The records also include model-input token information.