Source-linked AI summary
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, Hao Zhang
TL;DR
Real-world study of human interaction with LLMs is constrained by the lack of diverse queries across multiple models. The paper introduces LMSYS-Chat-1M and demonstrates its use across moderation, safety, instruction fine-tuning, and benchmark-question creation. These applications show the dataset’s utility for studying and advancing LLM capabilities, while its user distribution and data quality limit generalization.
Problem
Research on human-LLM interaction requires diverse real-user queries across models, but such datasets are difficult to obtain because collection is costly, proprietary, and hard to incentivize.
Method
The paper introduces LMSYS-Chat-1M, collected from a free online service serving 25 LLMs, and evaluates it through four representative use cases.
Results
The dataset supports content moderation, safety benchmarking, instruction-following model training, and challenging benchmark-question creation, including moderation performance on par with GPT-4.
Takeaways & Limitations
LMSYS-Chat-1M provides a resource for understanding user interactions with LLMs and for developing moderation, instruction-following, and benchmarking applications.
Takeaways & Limitations
The dataset may not generalize across all user groups because its users are predominantly LLM hobbyists and researchers.
Abstract
from arXiv · showhide
Studying how people interact with large language models (LLMs) in real-world scenarios is increasingly important due to their widespread use in various applications. In this paper, we introduce LMSYS-Chat-1M, a large-scale dataset containing one million real-world conversations with 25 state-of-the-art LLMs. This dataset is collected from 210K unique IP addresses in the wild on our Vicuna demo and Chatbot Arena website. We offer an overview of the dataset's content, including its curation process, basic statistics, and topic distribution, highlighting its diversity, originality, and scale. We demonstrate its versatility through four use cases: developing content moderation models that perform similarly to GPT-4, building a safety benchmark, training instruction-following models that perform similarly to Vicuna, and creating challenging benchmark questions. We believe that this dataset will serve as a valuable resource for understanding and advancing LLM capabilities. The dataset is publicly available at https://huggingface.co/datasets/lmsys/lmsys-chat-1m.
1 INTRODUCTION
The paper addresses the lack of diverse real-user queries across multiple LLMs by introducing LMSYS-Chat-1M and demonstrating four applications of the dataset.
- Motivation: The dataset is motivated by the need to study how users interact with LLMs, including their query patterns, expectations, trust, and safety-related questions.The paper connects these interaction patterns to improving LLMs’ ability to serve user needs and avoid misuse.
- Motivation: A central obstacle is that diverse real-user queries across different LLMs remain difficult to collect because hosting is costly, vendor data is proprietary, and open models attract limited participation.These constraints make large-scale multi-LLM conversation datasets difficult to create.
- Dataset contribution: LMSYS-Chat-1M is introduced as a large-scale, real-world conversation dataset collected from a free online service serving 25 LLMs.The service included both open-source and proprietary models and operated for five months.
- Dataset contribution: The dataset contains over 1 million user conversations spanning diverse languages and topics, with user consent and safety-oriented release measures.The authors removed personal identification information and flagged unsafe and toxic content while retaining original conversations.
- Use cases: The paper applies LMSYS-Chat-1M to content moderation, safety benchmarking, instruction fine-tuning, and challenging benchmark-question creation.These four use cases are presented as demonstrations of the dataset’s potential.
2 DATASET COLLECTION
The dataset was collected through three website chat interfaces and released as raw conversation text with privacy and moderation-related metadata or safeguards.
- Collection interfaces: Users could chat with one model, two randomly selected anonymous models, or two self-selected models through three website interfaces.The interfaces were Single model, Chatbot Arena battle, and Chatbot Arena side-by-side.
- Data handling: The dataset retains raw conversation text without processing while removing conversations containing personally identifiable information.OpenAI moderation API output is included for each message.
- Data handling: Unsafe conversations remain intact so researchers can study safety-related questions arising from real-world LLM use.The release combines privacy-oriented removal of PII with preservation of unsafe content.
- Dataset statistics: Table 1 compares conversation datasets using basic statistics, with token counts calculated by Llama2’s tokenizer.The caption defines “Conv” as conversation and “Lang” as language.
3 DATASET COMPOSITION
LMSYS-Chat-1M contains one million conversations from 25 state-of-the-art LLMs, 210K users, and more than 150 languages. Its composition spans multiple models, languages, and topics, with coding and software questions forming the largest topic group in the sampled analysis.
- 1 million conversations cover 25 state-of-the-art LLMs, 210K users, and more than 150 languages.
- Vicuna receives the most conversations because it is the website’s default model, while English, Portuguese, Russian, Chinese, and Spanish are the top five languages.
- Coding and software questions form the majority of questions in the 100K sampled English conversations.The analysis includes initial and follow-up user turns and displays results in Figure 3.
- The sampled topics also include unsafe content, general knowledge, business inquiries, and writing assistance.
- 5% of conversations are flagged as potentially harmful, although the moderation API may have low recall and miss additional harmful content.
4 USE CASES
The paper demonstrates four uses of LMSYS-Chat-1M: content moderation, safety benchmarking, instruction-following training, and challenging benchmark construction. Across these uses, the dataset supports model evaluation, fine-tuning, and analysis of real-world LLM behavior.
- Four use cases apply LMSYS-Chat-1M to content moderation, safety benchmarking, instruction-following training, and challenging benchmark questions.
- 4.1 DEVELOPING CONTENT MODERATION MODELS: 30% improvement was observed when fine-tuning Vicuna-7B into Vicuna-moderator-7B, which surpassed GPT-3.5-turbo and matched GPT-4 on the moderation benchmark.The benchmark contains toxic messages not flagged by the previous OpenAI moderation API version.
- 4.2 BUILDING A SAFETY BENCHMARK: LMSYS-Chat-1M contains jailbreak conversations that expose safety differences across models, including successful jailbreaks on proprietary models such as GPT-4 and Claude.Open models without safety-related training were reported as more vulnerable, while Llama-2 and Claude were safest in the benchmark.
- 4.3 TRAINING INSTRUCTION-FOLLOWING MODELS: HighQuality-7B was only slightly worse than Vicuna-7B, while Upvote-7B performed markedly worse than distilled counterparts in instruction-following evaluations.HighQuality used 45K conversations from OpenAI and Anthropic models; Upvote used 39K user-voted conversations from open models.
- 4.3 TRAINING INSTRUCTION-FOLLOWING MODELS: The HighQuality result suggests that LMSYS-Chat-1M prompts have quality similar to ShareGPT prompts used to train Vicuna.
5 LIMITATIONS
LMSYS-Chat-1M has several limitations affecting representativeness, data quality, and available annotations. These constraints matter for interpreting results and using the dataset fairly.
- Most website users are LLM hobbyists and researchers, so findings may not generalize across all user groups.Everyday users and people from other professions may interact with LLMs differently.
- The absence of registration and filtering can produce substantial amounts of duplicate and low-quality data.The authors intentionally retain these data to reflect the real-world distribution.
- The dataset contains raw conversations without human preference annotations.Although the website collects some user votes, their quality had not been examined for release.
6 RELATED WORK
LLM conversation research builds on earlier large-scale dialogue datasets but requires broader scale and topic coverage as LLM conversations become more comprehensive. LMSYS-Chat-1M differs by providing in-the-wild conversations with state-of-the-art LLMs.
- LLM conversations span broader and deeper topics, motivating datasets with greater scale and more diverse topic coverage.
- Comparable public resources include Anthropic Helpfulness and Harmlessness, OpenAssistant, and Chatbot Arena conversations.The paper discusses their differences from LMSYS-Chat-1M in Section 3.
- Unlike pre-LLM dialogue datasets such as UbuntuDialogue and MultiWOZ, LMSYS-Chat-1M contains in-the-wild conversations with state-of-the-art LLMs.
7 FUTURE WORK
The authors are considering quarterly dataset dumps to support transparency and accessibility as the LLM field evolves. This effort requires substantial resources and careful handling of privacy issues.
- Quarterly dataset releases are being considered to keep the resource current as the LLM field rapidly evolves.
- Maintaining quarterly releases would require considerable computing resources, maintenance effort, and user traffic.
- Future releases would need to handle potential data privacy issues carefully while seeking sponsors and collaborators.
8 CONCLUSION
The paper introduces LMSYS-Chat-1M, a one-million-conversation dataset for studying interactions with LLMs. It supports content moderation, instruction fine-tuning, and benchmarking as a resource for understanding and refining LLM technologies.
- LMSYS-Chat-1M contains one million LLM conversations and provides insights into user interactions with LLMs.
- The dataset supports content moderation, instruction fine-tuning, and benchmarking tasks.
- The authors present it as a resource for enhancing understanding and refinement of LLM technologies.
A DATA COLLECTION WEBSITE
The website provides both single-model chat and Chatbot Arena battle interfaces, while the moderation experiment defines categories and tests model classification behavior. The examples include harmful content and moderation outputs.
- The front end offers a single-model chat interface and a Chatbot Arena battle interface, built with Gradio.
- The moderation task classifies inputs into hate, self-harm, sexual, violence, and harassment categories, then requires an explanation and bracketed choice.
- Examples pair harmful user prompts with moderation labels, including violence and sexual-content cases.
B.3 EXAMPLES OF REFUSAL ON MODERATION TASK
The refusal examples show that some models decline harmful moderation-related requests while jailbreak prompts can elicit unsafe outputs from GPT-4. The paper identifies several techniques for bypassing safety measures.
- Some models, including Llama-2-7B and Claude-2, refuse to classify harmful prompts, whereas Vicuna-moderator-7B provides a category label.
- GPT-4 examples demonstrate that content warnings, educational framing, harmful rewrites, misspellings, token replacement, and translation can support jailbreaks.
B.5 GOOD PROMPTS FOR EVALUATION
The examples illustrate how real-user prompts can test problem solving, creativity, factual accuracy, and instruction following. Model comparisons favor GPT-4 on several challenging tasks.
- Challenging prompts combine multiple skills, such as mathematical reasoning, factual accuracy, creativity, and adherence to detailed formatting constraints.
- GPT-4 is judged better on the Carthage question because it supplies more facts and reasons, including General Hannibal, and gives 10 alternatives.
- GPT-4 correctly solves the fruit-ratio problem, while GPT-3.5-Turbo gives incorrect final quantities.
B.6 BAD PROMPTS FOR EVALUATION
The bad-prompt examples contrast simple or inaccessible requests with prompts designed to assess problem solving, creativity, and factual accuracy. The appendix also specifies a 1–10 scoring rubric and lists challenging prompt types.
- B.6 BAD PROMPTS FOR EVALUATION: A greeting is rated low because it requires no problem solving, creativity, or real-world factual adherence.
- B.6 BAD PROMPTS FOR EVALUATION: Requests for an IP address are poor evaluation prompts because the assistant lacks access to personal information and the task mainly tests factual accuracy.
- B.7 THE SYSTEM PROMPT FOR IDENTIFYING CHALLENGING QUESTIONS: The evaluation system asks models to assess problem solving, creativity, or factual accuracy and briefly explain each prompt’s potential.
- B.7 THE SYSTEM PROMPT FOR IDENTIFYING CHALLENGING QUESTIONS: Prompts receive scores from 1 to 10, with 8–10 high, 4–7 medium, and 1–3 low potential.
- B.7 THE SYSTEM PROMPT FOR IDENTIFYING CHALLENGING QUESTIONS: The rubric instructs evaluators to avoid high scores for ambiguous or overly straightforward prompts.
- B.7 THE SYSTEM PROMPT FOR IDENTIFYING CHALLENGING QUESTIONS: A ReactJS, ExpressJS, and TypeScript request receives score [[8]] because it tests problem solving and real-world factual adherence, but not creativity.
- B.8 EXAMPLES OF CHALLENGING PROMPTS FROM ARENA-HARD-200: Additional challenging examples include Perl code written in Shakespearean style, departmental SQL aggregation, sensitive-data replacement, and Swiss AI guidance.