Source-linked AI summary
OpenAssistant Conversations -- Democratizing Large Language Model Alignment
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, Alexander Mattick
TL;DR
Large-scale alignment research lacks openly available, broad, high-quality human feedback data. This paper releases a crowdsourced, human-annotated conversation dataset and finds that models trained on it consistently outperform corresponding baselines on standard benchmarks.
Problem
Open research lacks large-scale, broad, high-quality human feedback datasets for aligning language models with human preferences.
Method
The authors crowdsource and curate human-generated conversations with prompt, reply, quality, safety, and ranking annotations, then train SFT, reward, and RLHF models.
Results
Models trained on OpenAssistant Conversations consistently outperform corresponding baseline models on standard benchmarks, although RLHF surpasses SFT on some benchmarks but not others.
Takeaways & Limitations
The released dataset and models support more inclusive open research on large-language-model alignment.
Takeaways & Limitations
The dataset may reflect annotator demographic biases, as 89.1% of annotators identified as male and their perspectives may shape its values and interests.
Abstract
from arXiv · showhide
Aligning large language models (LLMs) with human preferences has proven to drastically improve usability and has driven rapid adoption as demonstrated by ChatGPT. Alignment techniques such as supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF) greatly reduce the required skill and domain knowledge to effectively harness the capabilities of LLMs, increasing their accessibility and utility across various domains. However, state-of-the-art alignment techniques like RLHF rely on high-quality human feedback data, which is expensive to create and often remains proprietary. In an effort to democratize research on large-scale alignment, we release OpenAssistant Conversations, a human-generated, human-annotated assistant-style conversation corpus consisting of 161,443 messages in 35 different languages, annotated with 461,292 quality ratings, resulting in over 10,000 complete and fully annotated conversation trees. The corpus is a product of a worldwide crowd-sourcing effort involving over 13,500 volunteers. Models trained on OpenAssistant Conversations show consistent improvements on standard benchmarks over respective base models. We release our code and data under a fully permissive licence.
1 Introduction
Large language models have advanced rapidly, but aligning them with human values and preferences remains difficult and heavily dependent on scarce, high-quality human feedback data. The paper addresses this access gap through an effort to democratize large-scale alignment research.
- Motivation: AI progress has largely followed scaling Transformer architectures, training data, and compute.The passage also notes that models can fit training data and generalize, though the supplied text is truncated.
- Motivation: Alignment seeks to make AI systems optimize intended objectives while adhering to human values, intentions, preferences, safety, and ethical standards.Assistant-style fine-tuning is presented as one possible approach.
- Motivation: ChatGPT illustrates assistant-style alignment through human-generated usage examples and reinforcement learning from human feedback.In RLHF, humans provide feedback as rewards or penalties.
- Problem: Large-scale alignment depends on high-quality data, yet open human-feedback datasets remain scarce and are often replaced by limited synthetic instruction data.The passage identifies limitations in synthetic datasets’ complexity, creativity, and quality.
- Contribution: Access to the resources required for large-scale training and data collection is concentrated in a few research labs, motivating efforts to democratize alignment research.The paper frames this concentration as undermining inclusive and diverse research on alignment challenges.
2 Data Format
The dataset is organized as Conversation Trees, whose nodes represent written messages and whose root is an initial prompt. Nodes are role-labelled and may branch into separate next steps, with each root-to-node path forming a valid thread.
- Data structure: Conversation Trees are the dataset’s basic data structure, with nodes representing written messages.Each tree is organized around a root node and subsequent message nodes.
- Data structure: The root node represents an initial prompt given by the prompter.The terminology distinguishes the prompter and assistant roles from human contributors, who are called users.
- Data structure: 12 messages form an example Conversation Tree of depth 4, and every root-to-node path is a valid thread.Nodes are labelled by role and can have multiple children of the opposite role, representing separate next steps.
3 Data Collection
Data collection used a crowdsourced web application that decomposed conversation-tree construction into discrete tasks, producing over 10,000 fully annotated and filtered trees from more than 625,000 volunteer tasks. The pipeline combined prompt and reply creation, message labeling, reply ranking, and state-based tree management to support efficient, diverse, and quality-controlled data collection.
- Collection workflow: More than 13,000 volunteers used a web application to collect conversation trees through prompting, labeling, replying, and ranking steps.Users were informed that their contributions would form part of a public dataset.
- Collection workflow: Over 625,000 completed tasks produced over 10,000 fully annotated and filtered Conversation Trees.The platform also tracked collection parameters, including the number of replies collected for each parent message.
- Task types: The task system included creating prompts, replying as assistants or prompters, labeling messages, and ranking two or more assistant replies.Assistant replies required careful consideration and often external research, while prompter replies emphasized diversity rather than strict quality.
- Annotation: Message labels covered spam detection, guideline adherence, and five-point Likert ratings for quality-related dimensions.Spam flags could trigger automatic removal, guideline violations were reviewed by moderators, and quality dimensions included creativity, humorousness, politeness, and harmlessness.
- Tree management: A tree state machine advanced each conversation tree through initial prompt review, growing, and end states to manage task progression and data collection.The growing state accumulated replies, labels, rankings, and multiple conversation paths, while platform parameters controlled collection behavior.
4 Dataset Composition
The dataset contains 161,443 messages across 66,497 conversation trees and 35 languages, with 461,292 quality ratings and multiple released dataset variants. Human annotations include Likert-scale quality dimensions, binary safety-related labels, and assistant-message preference rankings.
- Dataset scale: 161,443 messages span 66,497 conversation trees and 35 languages, with 461,292 quality ratings.The messages comprise 91,829 prompter messages, 69,614 assistant messages, and 8,576 synthetic messages, leaving 152,867 human-submitted messages.
- Dataset scale: 10,968 conversation trees are complete, while 52,159 are incomplete.Complete trees have had their full number of messages collected and their moderation process concluded.
- Annotations: Human labels cover six Likert-scale categories and five binary categories, alongside rankings of assistant messages for the same prompt.Likert categories are Creativity, Quality, Humor, Helpfulness, Violence, and Rudeness; binary categories include Language Mismatch, Not Appropriate, Personally Identifiable Information, Hate Speech, and Sexual Content.
- Language and participation: English and Spanish messages dominate the dataset, and contributions include a small number of power users.The text attributes English prominence to the English-speaking open-source machine learning community and Spanish volume to publicity from the Spanish machine learning community.
- Release variants: The dataset is released in variants containing initial prompts, completed trees, all trees, or messages filtered for spam.The completed-tree variant is recommended for most purposes, while non-completed trees can be included when more data is required.
5 Contributor Demographics and Satisfaction
A Discord-announced survey collected 270 responses on contributor demographics, motivations, and satisfaction. Over 95% were glad to contribute, while about 40% were participating in an open-source project for the first time, although the sample was recruitment-biased.
- Survey scope: 270 survey responses covered contributor demographics, personal motivation, and user satisfaction.The survey was distributed through an announcement on the project’s Discord channel.
- Satisfaction: Over 95% of respondents agreed or strongly agreed that they were glad to contribute to OpenAssistant.This result combines respondents selecting either agreement category for the overall-contribution statement.
- Contributor experience: About 40% reported that contributing to OpenAssistant was their first open-source project.The survey result reflects respondents’ self-reported prior open-source experience.
- Limitations: Discord recruitment biased the sample toward platform users active near the announcement, and active contributors were more likely to respond.These limitations may affect how representative the survey responses are of all contributors.
6 Experimental Validation
The experiments evaluate SFT, reward-model, and RLHF models based on Pythia, LLaMA, and Falcon using standard benchmarks, and assess automated toxicity detection against human labels and moderation outcomes. Automated and human toxicity labels show agreement, while moderation removes messages rated more toxic across all six categories, though toxicity scores alone are insufficient for exclusion decisions.
- Model evaluation: The study trains supervised fine-tuned, reward, and reinforcement-learned models based on Pythia, LLaMA, and Falcon.Pythia and Falcon have permissive open-source licenses, whereas LLaMA has a bespoke non-commercial license.
- Model evaluation: Evaluation uses standard benchmarks, with LMEH aggregating performance across BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA.Evaluations were performed externally using FastEval, with results hosted on a continually updated leaderboard.
- Toxicity analysis: 115,153 messages are used to correlate automated Detoxify ratings with human toxicity labels, showing agreement in at least one element of every correlation-matrix row and column.The automated detector covers English, Spanish, Russian, French, and Italian and distinguishes six toxicity categories.
- Toxicity analysis: All six Detoxify toxicity categories have markedly higher values for deleted messages than retained messages, indicating effective moderation.Deleted messages either failed community moderation or were manually removed, while retained messages remained in the dataset.
- Toxicity analysis: Average toxicity values remain small even for deleted messages, so Detoxify ratings alone cannot determine whether messages should be excluded.Non-toxic messages may be deleted for factual inaccuracies or poor grammar, and descendants of deleted messages may also be affected.
7 Limitations
The project’s open data-collection approach creates limitations involving reward-model training, demographic bias, and potentially unsafe residual content. Accordingly, the authors recommend academic-research use only and careful safety and bias evaluation before downstream deployment.
- Reward model data collection: Reward models are trained on rankings of human-generated messages rather than messages generated by the initial SFT model used in InstructGPT.This choice reflects the project’s existing collection of ranking data for quality control, spam filtering, and dataset sub-sampling.
- Subjective, Cultural, and Contribution Frequency Biases: 89.1% of annotators identify as male, with a median age of 26, creating potential subjective, cultural, and contribution-frequency biases.The project’s open nature makes dataset-bias control difficult because annotator demographics are heterogeneous in some dimensions but homogeneous in others.
- Possibility of Unsafe Content: Residual unsafe content may remain despite measures to detect and remove harmful messages, requiring careful evaluation of trained models.The authors view transparent filtering as a path toward higher standards but acknowledge that the system is not infallible.
- Recommendations: The authors advocate using the released models only in academic research and investigating safety and bias before downstream tasks.They warn that the models may exhibit unsafe behavior and are likely susceptible to prompt injection attacks.
8 Safety and Ethical Implications … B Ranking merging
The paper frames OpenAssistant as a transparency contribution addressing the societal risks of powerful language models, while its guidelines and ranking-merging procedure formalize safer data collection and preference aggregation.
- 8 Safety and Ethical Implications: LLMs can hallucinate, generate toxic or hateful content, violate constraints, and reproduce biases, while RLHF may mitigate some shortcomings but exacerbate others.These risks motivate explicit safety and ethical considerations in alignment work.
- 8 Safety and Ethical Implications: The authors argue that powerful language models can significantly affect society, making transparency in development and deployment essential.They present OpenAssistant Conversations as their contribution to this transparency goal.
- A Contributor Guidelines: Contributor guidelines require polite, friendly, readable, and consistently styled assistant replies that prioritize verified information, factual accuracy, and helpfulness.Contributors are instructed to flag unverified information and provide warnings for high-risk topics such as medicine, law, or chemistry.
- A Contributor Guidelines: Replies are ranked by how well they satisfy the user’s request and follow the guidelines, with factual accuracy and helpfulness first, and inadequate warnings or poor readability penalized.An admission of ignorance ranks below a factually correct reply but above a factually incorrect one.
- B Ranking merging: RLHF requires merging rankings from multiple human rankers because individual preferences have high variance and may not represent a generally capable assistant.The goal is to produce a preference structure aligned with users rather than relying on isolated judgments.
- B Ranking merging: Ranked pairs, or Tideman’s method, fuses preferences as a ranked-voting problem by ordering winners according to the strength of pairwise preferences.The resulting directed acyclic graph is converted into a preference list by recursively removing source nodes and appending them to the back.
C Web application for data collection … G Collection parameters
The paper describes a crowdsourcing interface for labeling, replying, and ranking, supplemented by safeguards against ChatGPT-generated inputs and user feedback surveys. It also summarizes topic diversity and configurable parameters governing collection, review, moderation, and acceptance.
- C Web application for data collection: The web platform supported labeling, assistant reply input, and ranking tasks, with heuristic-based random task selection to increase variation and engagement.The interface evolved during the project, so early users encountered a less refined experience.
- C Web application for data collection: Labeling presented conversation trees up to a highlighted message for evaluation with predefined questions, while ranking used drag-and-drop comparisons and allowed factual-incorrect markings.These functions were shown in the platform previews.
- D Filtering ChatGPT Inputs: The collection guidelines discouraged copying AI-generated responses, banned users posting ChatGPT-generated content, deleted their contributions, and used automatic tests to detect cases.ChatGPT’s widespread use during collection created a risk that some inputs were not fully human-generated.
- F Word Cloud: The English subset was analyzed with 40 LDA models, selecting 33 topics at peak coherence and revealing variety from biology, cooking, and music to quantum physics.The resulting topic word clouds are presented in Figure 11.
- G Collection parameters: Collection parameters controlled active trees, initial-prompt review capacity, tree depth, child counts, replies per assistant response, target tree size, peer reviews, and acceptance thresholds.These settings regulate how many messages and reviews are gathered and when prompts or replies are accepted.
- G Collection parameters: Additional parameters enabled auto-moderation, handled skipped or red-flagged messages, set full-labeling probabilities, and managed ranking activation, participation, and backlog behavior.The parameter definitions also specify review counts and conditions for activating ranking tasks or halting and aborting trees.
H Training Configuration
The training pipeline uses Pythia and LLaMA base models for supervised fine-tuning, reward modeling, and PPO optimization. It formats conversation threads with special role and end-of-text tokens, trains SFT only on assistant replies, and stabilizes PPO with a KL penalty.
- The pipeline trains supervised fine-tuned models, reward models, and PPO-fine-tuned models based on reward-model predictions, using decoder-only Pythia and LLaMA base models.
- Conversation format: Conversation threads are serialized by alternating prompter and assistant tokens, with end-of-text markers separating prompts and replies.Prompts and replies may each contain multiple tokens after tokenization.
- Supervised fine-tuning: SFT masks prompt tokens and trains the model to predict only assistant-reply tokens from conversational data.
- Reward model: The reward model replaces the language-modeling head with a linear layer producing a score for the last reply and learns from ranked replies to the same prompt.It evaluates performance by predicting the better reply among differently ranked pairs on held-out validation data.
- PPO training: PPO fine-tunes the SFT model on reward-scored replies to unanswered questions while applying a per-token KL penalty to prevent instability and reward over-optimization.Training uses the trlx framework.
I Evaluation Tasks
The evaluation uses standardized language-model benchmarks from lm-evaluation-harness, alongside Vicuna Elo Rank, OpenAI Evals, and HumanEval. The lm-evaluation-harness subset comprises BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA.
- lm-evaluation-harness: lm-evaluation-harness evaluates language models on standardized tasks, including BoolQ, PIQA, HellaSwag, WinoGrande, ARC-e, ARC-c, and OBQA.The selected tasks are a subset of the framework’s available evaluations.
- Vicuna Elo Rank: Vicuna Elo Rank uses LLMs to judge other LLMs and assign rankings.
- OpenAI Evals: OpenAI Evals is a set of general LLM evaluation benchmarks released with GPT-4.OpenAI also invited the community to contribute to the benchmark set.
- HumanEval: HumanEval measures functional correctness when synthesizing programs from docstrings.OpenAI released HumanEval with the announcement of Codex.
J Political Compass Evaluations · K Community Engagement · M Author Statement
Preliminary evaluations suggest that OpenAssistant-trained models exhibit more balanced and varied political leanings than ChatGPT, while the project’s global community expanded through Discord, GitHub, and media engagement. The authors accept responsibility for rights violations and commit to addressing problematic data when necessary.
- J Political Compass Evaluations: The evaluation used standardized prompts, multiple samples, and majority voting to assess an OpenAssistant-fine-tuned model on political-leaning tests.The authors emphasize that these results are preliminary because no consensus exists on the methodology for political-leaning evaluations.
- J Political Compass Evaluations: Preliminary results indicate that the OpenAssistant-trained model has more balanced and varied political leanings than ChatGPT.The authors caution against interpreting this comparison with high certainty and plan to update the section with improved results.
- K Community Engagement: The project built a large global community, including an active Discord group and a GitHub repository with over 200 contributors.Community growth was documented through Discord membership, GitHub commits and stars, and YouTube video views over time.
- K Community Engagement: The figures underscore what collective volunteer effort can achieve in a field previously dominated by a small number of industrial labs.This frames community participation as a way to broaden involvement in large-language-model research.
- K Community Engagement: Comparing contributor and subscriber growth with OpenAssistant-themed videos suggests that media events influenced the project’s development.The passage links the influx of participants with the emergence of themed videos, without quantifying the relationship.
- M Author Statement: The authors accept responsibility for rights violations arising during data collection or other project work.They state that appropriate action will be taken when needed, including removing data associated with such issues.