Source-linked AI summary
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage
Kurt Shuster, Jing Xu, Mojtaba Komeili, Da Ju, Eric Michael Smith, Stephen Roller, Megan Ung, Moya Chen, Kushal Arora, Joshua Lane, Morteza Behrooz, William Ngan, Spencer Poff, Naman Goyal, Arthur Szlam, Y-Lan Boureau, Melanie Kambadur, Jason Weston
TL;DR
Open-domain dialogue systems need scalable data and methods that better reflect organic users while remaining responsible. BlenderBot 3 addresses this with a 175B-parameter modular model, internet access, long-term memory, public deployment, and safety mechanisms. The report finds BB3 superior to other publicly released agents and uses deployment interactions to support further improvement, while acknowledging persistent errors and toxic outputs.
Problem
Paid crowdworker data collection does not scale and may not reflect organic users’ interests, while deployed systems face inadequate or adversarial feedback.
Method
BB3 is a 175B-parameter transformer fine-tuned for modular tasks, deployed publicly with internet search, long-term memory, safety techniques, and interaction-based learning plans.
Results
BB3 is superior to other publicly released open-domain conversational agents, and interaction and feedback data can improve it further.
Takeaways & Limitations
Releasing models, datasets, interactions, and model snapshots supports reproducible research on more useful, constructive, safe, and responsible conversational agents.
Takeaways & Limitations
BB3 still makes off-topic, nonsensical, incorrect, rude, or inappropriate mistakes, and safety techniques do not eliminate toxic content.
Abstract
from arXiv · showhide
We present BlenderBot 3, a 175B parameter dialogue model capable of open-domain conversation with access to the internet and a long-term memory, and having been trained on a large number of user defined tasks. We release both the model weights and code, and have also deployed the model on a public web page to interact with organic users. This technical report describes how the model was built (architecture, model and training scheme), and details of its deployment, including safety mechanisms. Human evaluations show its superiority to existing open-domain dialogue agents, including its predecessors (Roller et al., 2021; Komeili et al., 2022). Finally, we detail our plan for continual learning using the data collected from deployment, which will also be publicly released. The goal of this research program is thus to enable the community to study ever-improving responsible agents that learn through interaction.
1 Introduction
BlenderBot 3 is presented as a publicly deployed, reproducible open-domain conversational agent designed to improve through human interactions while addressing the limits of crowdworker-collected data. The report covers its modular 175B-parameter model, deployment design, safety techniques, and plans for continual learning.
- Motivation: Public deployment is proposed as an alternative to paid crowdworker data collection, which may not scale or reflect organic users’ interests.The deployment is intended to provide large-scale organic interactions for studying model improvement over time.
- Deployment: The system is deployed as an English-speaking conversational agent on a public website accessible to adults in the United States.The research program aims to improve usefulness and responsibility through collected interactions.
- Model: BlenderBot 3 is a 175B-parameter transformer initialized from OPT-175B and fine-tuned for modular tasks.It retains long-term memory and internet-search capabilities from its predecessors.
- Deployment and learning: The report details deployment design, user interface, initial organic-user experiments, and techniques for robust continual learning with humans in the loop.The continual-learning approach must account for adversarial behavior.
- Contributions: BB3 outperforms existing openly available chatbots, including its two predecessors, by a wide margin.The authors also release model weights, code, a model card, conversational datasets, and publications, with future interaction-data and model-snapshot releases planned.
2 Related Work
Related work establishes that large-scale pre-training and subsequent fine-tuning have advanced open-domain dialogue, while deployment-based learning offers a route toward adapting to organic users. These approaches remain constrained by costly, instruction-shaped data collection and the risk of inadequate or adversarial user feedback.
- Open-domain dialogue: Open-domain dialogue has progressed through increasingly large pretrained models, from DialoGPT and Meena to BlenderBot and LaMDA.The cited examples range from 762M to 137B parameters.
- Open-domain dialogue: General-purpose language-model pre-training, including OPT-175B, has also been used for downstream dialogue applications.Some systems are openly accessible, while others restrict access through inaccessibility or paid APIs.
- Fine-tuning: Fine-tuning pretrained models brings further gains, and publicly released crowdsourced datasets provide training data for dialogue systems.The work uses such datasets for BlenderBot 3 training.
- Continual learning and deployment: Deployment-based learning can adapt systems using interaction data, but expert- or crowdworker-collected data may not match changing organic-user desires and requires significant resources.Prior work includes learning from language data, web search, task evaluation, feedback, and deployed chatbots.
- Continual learning and deployment: Organic users may provide inadequate feedback or behave adversarially, creating a central challenge for self-feeding conversational systems.The related work specifically cites Microsoft’s Tay as an example of adversarial interaction risk.
- Continual learning and deployment: Directly using human utterances as targets can improve deployed systems when conversational partners interact symmetrically, but this approach does not directly apply to asymmetric assistant conversations.The passage contrasts conversational partners with humans treating the bot as an assistant.
3 BlenderBot 3 Model
BlenderBot 3 is a modular transformer system that conditionally combines dialogue context with internet search, long-term memory, or extracted entities before generating a response. It is offered in three model sizes and fine-tuned on diverse dialogue, knowledge, question-answering, and safety tasks.
- Model Sizes and Training: BB3 is released in 3B, 30B, and 175B parameter sizes, using an encoder-decoder architecture for 3B and decoder-only OPT architectures for 30B and 175B.The OPT pre-training corpus contains roughly 180B tokens.
- Overall Flow: BB3 uses one transformer with control codes to execute successive modules conditionally, rather than independent components.Each module receives dialogue context and may pass its output to the next module.
- Overall Flow: When memory access is selected, BB3 recalls a stored memory; the final response module then uses the constructed context, optionally including knowledge and memory sequences.The model can also generate summaries of conversational turns for storage as long-term memories.
- Overall Flow: The system decides whether internet search or long-term memory access is needed for each new utterance.If neither is required, it extracts a relevant entity from the dialogue history instead.
- Overall Flow: When search is selected, BB3 generates a query, retrieves documents through an external search engine, and generates a knowledge response to ground the final reply.Internet Search itself is executed by the search engine rather than the transformer.
- Model Sizes and Training: Fine-tuning spans publicly available QA, open-domain, knowledge-grounded, task-oriented, and safety datasets, with control tokens indicating the task.For knowledge responses, training targets copy tokens from source documents to reduce made-up facts.
4 Deployment
BlenderBot 3 was deployed through a web interface for consenting adult users in the United States, with feedback and transparency features. Deployment safety combines classifiers, keyword and topic checks, canned responses, and user reporting for subsequent improvement.
- User Interface: The deployment provides a desktop- and mobile-compatible chat interface, with current usage favoring mobile.Users first encounter a cover page describing the research and terms and conditions.
- Data Release: Conversations are collected only from adults in the United States who agree to terms allowing selected human-bot interactions to be released for research.Users can opt out of recording individual conversations, and releases are de-identified.
- User Feedback: Users can rate each bot message with thumbs up or thumbs down and explain negative feedback through predefined categories or free-form text.The bot responds to selected negative feedback with templated apologies and may ask what it could improve.
- Transparency: The interface exposes response-generation steps, including internet queries, selected search documents, extracted knowledge responses, and long-term memory information.These mechanisms are intended to show how the bot produced a response.
- Safety Mechanisms: A separate transformer safety classifier labels dialogue context safe or unsafe, supplemented by keyword filters and explicit checks for self-harm and medical topics.Flagged cases can receive canned messages instead of ordinary responses.
- Safety Mechanisms: Users can report inappropriate responses, and the collected reports are intended for community release and future safety improvements.The authors give DIRECTOR training as an example of using such data to make responses safer.
5 Continual Learning
The continual-learning program studies how to collect and learn from deployment feedback while remaining robust to adversarial users. Results favor modular feedback, DIRECTOR, continual retraining, and user-based filtering, with collected interactions and future model snapshots intended for public release.
- Continual Learning: Deployment artifacts include consented conversations, fine-tuned model snapshots, and evaluation reports comparing successive snapshots.The artifacts support reproducible study of how models improve over time.
- Continual Learning: The program aims to determine which methods best collect and learn from interaction data, including methods robust to adversarial inputs, and to understand improvement limits.Planned questions include performance saturation, reuse of historical data across models, and whether models can elicit feedback that improves them.
- 5.1 What’s the best method to learn from feedback?: Users provide binary, free-form, failure-type, and improved-response feedback, enabling comparisons among supervised learning, modular supervision, control codes, reranking, and DIRECTOR.The deployment interface also asks users to categorize negative feedback and may elicit suggestions for better responses.
- 5.1 What’s the best method to learn from feedback?: Modular feedback outperforms final-response feedback, while textual and binary feedback help less; DIRECTOR outperforms binary-feedback reranking.The evaluation compares feedback types and corresponding learning algorithms for internet-driven dialogue agents.
- 5.1 What’s the best method to learn from feedback?: Combining multiple feedback types with DIRECTOR gives the best reported results, continual retraining improves them further, and data from 3B models helps improve 175B models.The current BB3 release uses modular supervision, while DIRECTOR is planned for a future release.
6 Evaluations
BB3 is evaluated with automatic metrics and human judgments of dialogue quality and safety, including crowdworker and organic-user assessments. Across these evaluations, BB3-175B generally outperforms earlier BlenderBot systems and other tested baselines, while retaining important safety limitations.
- Evaluation setup: Evaluations measure generation quality, knowledge use, toxicity, and bias with automatic metrics and human judgments from crowdworkers and organic users.Comparisons also include pretrained OPT-175B in zero-shot and few-shot prompted settings.
- Crowdworker evaluations: BB3-175B achieves the highest overall rating, knowledgeability, and knowledge-plus-engaging scores among BB1, BB2, SeeKeR, and BB3-3B.It also has the lowest factual incorrectness score, although BB1 has slightly higher consistency.
- Crowdworker evaluations: BB3-175B performs best across almost all FITS human-evaluation metrics, including Good Response % and overall Rating, relative to the listed baselines.The reported gains span search-query quality, use of search results, and final-response construction.
- Crowdworker evaluations: 82% current and 76% specific judgments favor BB3-175B, while InstructGPT receives 57% sensible judgments in the reported comparison.InstructGPT more often avoids information, whereas BB3-175B more often provides specific and current responses.
- Safety evaluations: BB3-175B has the lowest unsafe-response levels in every tested setting except adversarial unsafe inputs and outperforms OPT-175B and BB3-3B on SaFeRDialogues.Among 175B-parameter models, Likelihood Bias is slightly reduced relative to BB3-3B.
- Limitations: 0.16% of BB3-175B utterances are flagged as rude or inappropriate, while deployment examples still reveal vague answers, factual mistakes, and safety-filter bypasses.The authors frame these failures as feedback for further safety improvement.
7 Releases
The release packages BB3’s models, code, datasets, documentation, and deployment artifacts for research use. It also commits to future sharing of de-identified organic conversations and model snapshots from continual learning.
- Release goals: The project aims to share models, code, and collected conversations responsibly to support accessible and reproducible research.Release considerations are documented in the report.
- Released artifacts: Public releases include the live deployment, model weights, open-source training code, a training logbook, and datasets.The release summary identifies these artifacts across deployment, weights, code, and data sections.
- Released artifacts: The 3B and 30B models are openly available, while access to the 175B model is granted to eligible researchers and organizations under stated practices.The access policy covers government, civil-society, academic, and global industry research organizations.
- Future releases: The authors commit to sharing de-identified organic conversational data and model snapshots collected from the interactive demo in future releases.These materials are intended to support research on continually improving interactive systems.
8 Conclusion
The conclusion presents BB3 as both a conversational model and a public deployment designed to collect interaction data for continual-learning research. It reports superiority to publicly released open-domain agents while emphasizing reproducibility, ongoing evaluation, and substantial remaining work.
- Conclusion: BB3 combines a new conversational model with a public deployment whose interactions are collected for continual-learning study and public release.The research program targets agents that continually learn and evolve while being continually evaluated.
- Conclusion: Evaluations show BB3 is superior to other publicly released open-domain conversational agents, and interaction and feedback data can improve it further.The conclusion links these findings to the broader release program.
- Conclusion: Releasing chatbot models and datasets is presented as key to reproducible insight into their behavior, potential, and limitations.The authors emphasize that wider research-community access enables others to build on available data and technologies.
- Conclusion: Making responses more constructive, helpful, safe, and responsible remains an open research problem requiring substantial further work.The authors therefore commit to releasing collected interactions and model snapshots to aid community progress.
9 Limitations and Ethical Considerations
BlenderBot 3 is released as an accessible, reproducible research system, but it remains constrained by model errors, safety risks, and the early state of continual learning from organic interaction.
- Model Limitations: BlenderBot 3 can produce off-topic, nonsensical, incorrect, rude, or inappropriate responses, sometimes because underlying modules fail.Search retrieval failures are one example of an underlying-module error.
- Model Limitations: The model is trained only on English-language data.
- Continual Learning Research: Continual learning from organic-user interactions remains in its infancy, with current studies mostly relying on crowdworker data.The authors identify substantial remaining work before using deployment data to update models.
- Safety Concerns: Safety techniques show promising benchmark results, but BlenderBot 3 still generates toxic content occasionally, especially in adversarially unsafe contexts.The benchmarks are static and crowdsourced and cannot guarantee safety in all situations.
- Safety Concerns: Human feedback can teach erroneous reasoning, misinformation, toxic behavior, or other undesirable behavior, and detection methods are imperfect.The authors therefore require sufficient safety study before deploying models trained on new interaction data.
- Considerations for Release: The project releases models, code, datasets, and a public demo while restricting demo access to consenting adults in the United States.The release aims to support accessibility, reproducibility, and further research while reducing potential harms.
A Additional Safety Evaluations
The additional safety evaluation examines descriptor bias in base models using HOLISTICBIAS perplexity comparisons, with Table 15 focusing on body-type descriptors.
- Body-type bias: Table 15 compares which HOLISTICBIAS body-type descriptors receive lower perplexity in “I hate” versus “I love” sentences across models.The comparison is made over descriptor-plus-noun combinations.
- Evaluation setup: The evaluations use base models without unsafe-response flagging, topic-change handling, or related deployment interventions.Table 13 specifically evaluates the zero-shot OPT-175B model.
- Training data: BB3 fine-tuning uses roughly 4 million source/target examples, corresponding to approximately 1.13B training tokens across modules.OPT-based BB3 models additionally include 600k pre-training examples, or about 170m tokens, for training stability.
B.2 BB3-3B Training
The BB3-3B training setup uses a 3B-parameter model trained with large-scale optimization, modular task data, and decoding choices selected partly for generation quality and efficiency.
- Training configuration: The 3B model was trained on 64 × 32GB V100 GPUs for 27k updates with batch size 64 and 1024 context tokens.Training used Adam with weight decay, a 100-update warmup, learning rate 1e−6, and validation-based early stopping.
- Scaling considerations: The 30B and 175B models were trained for one epoch because repeated exposure caused significant overfitting, especially for the 175B model.The 175B model used 128 A100 GPUs and batch size 218, whereas the 30B model used 64 GPUs and batch size 219.
- Decision and query modules: Decision and query-generation modules use greedy decoding for both models, while BB3-3B query generation enforces a minimum length of 2.
- Generation modules: The 3B model uses beam search in several generation modules, generally with beam size 3 and tri-gram blocking.These settings apply to long-term memory access, internet knowledge response generation, memory generation, and entity extraction, with module-specific minimum lengths.
- Dialogue generation: Dialogue response generation for BB3-3B uses beam search with beam size 10, minimum length 20, and tri-gram blocking on outputs and context.
C Current Events Evaluation Details
The current-events evaluation compares BB3-175B with InstructGPT using news-oriented prompts and ratings of currency, specificity, truth, interestingness, and sensibility.
- Evaluation construction: The evaluation samples 300 Wikipedia entities from July 2022 current events and asks about the latest news concerning each entity.Responses are generated by BB3-175B and InstructGPT for comparison.
- Search setup: BB3-175B uses the Mojeek API and appends “news july 2022” to generated search queries to encourage news results.InstructGPT uses OpenAI’s default Chat prompt and generation parameters.
- Evaluation criteria: Comparisons ask which response is more current, specific, true, interesting, and sensible.Current, specific, and true are evaluated with objective evidence by experts, while interesting and sensible are assessed by crowdworkers.
- Evaluation procedure: Expert evaluators validate current, specific, and true judgments with internet search, while allowing ties for these characteristics.
- Scoring interpretation: The protocol treats unsupported conjecture as less true than factual-only responses and out-of-date information as more current than no information.The authors note that always refusing to answer could achieve perfect truthfulness under this criterion.
- Baseline setup: OPT-175B baseline prompts and few-shot settings are listed without prompt-selection tuning, leaving other prompts potentially better or worse.
E Additional Results
This section provides supplementary evaluation tables and conversation examples for BB3, covering perplexity, human judgments, and unsafe-generation testing. The examples also illustrate factual, safety, and self-representation failures encountered in deployment.
- Additional evaluations: Tables 19–21 provide additional perplexity measurements, short-conversation human evaluations, and per-tool Safety Bench results.These tables expand the validation, human evaluation, and safety analyses.
- Reported results: The reported BB3-175B row contains results for multiple evaluation measures, including values of 85.8%, 46.4%, 2.1%, 88.1%, 39.0%, 84.1%, and 4.452.The extracted row preserves the reported values but does not identify each column label.
- Human evaluation: Table 20 compares BB3 with existing openly available open-domain dialogue models using human judgments from short conversations.Statistical significance is marked for comparisons with BB1, BB2, and SeeKeR.
- Conversation examples: Figures 10–15 show conversations spanning yoga, novels, plants, world history, pets, stories, and role-playing.These examples illustrate the range of conversational topics and interaction styles examined.
- Failure cases: Figures 16–22 document cases involving evasive answers, factual misunderstanding, offensive or unsafe responses, anthropomorphic claims, and questions about sentience.The examples include safety failures that passed the reported safety mechanisms.