Source-linked AI summary
Enhancing Chat Language Models by Scaling High-quality Instructional Conversations
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, Bowen Zhou
TL;DR
Existing approaches had not surpassed Vicuna, leaving room to improve open-source chat models through higher-quality, more diverse instructional data. The paper constructs UltraChat around broad human-AI interactions and fine-tunes LLaMA-13B to create UltraLLaMA. Evaluations report consistent outperformance of other open-source models, while the dataset is English-only and ChatGPT-based evaluation is less reliable than GPT-4.
Problem
Existing approaches had not produced open-source chat models surpassing Vicuna, motivating improved instructional data for raising performance.
Method
The paper constructs UltraChat from three broad interaction sectors and fine-tunes LLaMA-13B on it to create UltraLLaMA.
Results
UltraLLaMA consistently outperforms previous open-source models across the paper’s evaluations.
Takeaways & Limitations
UltraChat is presented as a leading open-source instructional dataset, while UltraLLaMA advances conversational performance over prior open-source models.
Takeaways & Limitations
UltraChat supports only English, and ChatGPT-based automatic evaluation is less reliable than GPT-4.
Abstract
from arXiv · showhide
Fine-tuning on instruction data has been widely validated as an effective practice for implementing chat language models like ChatGPT. Scaling the diversity and quality of such data, although straightforward, stands a great chance of leading to improved performance. This paper aims to improve the upper bound of open-source models further. We first provide a systematically designed, diverse, informative, large-scale dataset of instructional conversations, UltraChat, which does not involve human queries. Our objective is to capture the breadth of interactions that a human might have with an AI assistant and employs a comprehensive framework to generate multi-turn conversation iteratively. UltraChat contains 1.5 million high-quality multi-turn dialogues and covers a wide range of topics and instructions. Our statistical analysis of UltraChat reveals its superiority in various key metrics, including scale, average length, diversity, coherence, etc., solidifying its position as a leading open-source dataset. Building upon UltraChat, we fine-tune a LLaMA model to create a powerful conversational model, UltraLLaMA. Our evaluations indicate that UltraLLaMA consistently outperforms other open-source models, including Vicuna, the previously recognized state-of-the-art open-source model. The dataset and the model will be publicly released\footnote{\url{https://github.com/thunlp/UltraChat}}.
1 Introduction
Open-source chat models had not surpassed Vicuna, motivating better training data. The paper introduces UltraChat and UltraLLaMA, whose evaluations show consistent gains over open-source baselines.
- Open-source instruction-tuned chat models had not surpassed Vicuna, despite promising performance from existing approaches.
- UltraChat scales instructional conversations by targeting broad human-AI interactions rather than specific tasks.Its construction uses three interaction sectors, meta-information, expansion, and iterative prompting.
- UltraLLaMA is produced by fine-tuning LLaMA-13B on UltraChat and is evaluated against numerous open-source baselines.
- UltraLLaMA consistently outperforms open-source baselines in ChatGPT-based scoring and preference comparisons.The paper reports the highest independently assessed scores and consistent preference-study wins.
2 Related Work
Prior work established instruction tuning and LLM-based data augmentation as effective routes to capable chat models. UltraChat builds on this direction while emphasizing large-scale, multi-turn conversational data.
- Instruction tuning improves instruction understanding and generalization to unseen tasks.Prior work scaled instruction tuning from 60 NLP datasets to 1,836 tasks.
- LLM-based data augmentation offers a more feasible alternative to collecting large volumes of human-annotated instruction data.Alpaca generated 52k instruction-response pairs from 175 seed tasks using Text-Davinci-003.
- UltraChat’s construction process derives its three data sectors from different meta-information sources.
- Existing augmentation efforts include role-play and other prompt-engineering approaches for producing conversational instruction data.CAMEL generated 115k instruction-response pairs through a multi-agent role-play environment.
3 Design
UltraChat is designed to represent broad human-AI information-seeking interactions through three complementary sectors. Its construction emphasizes informative multi-turn conversations and comprehensive coverage.
- Directly generating multi-turn conversations can be satisfactory but not sufficiently informative, motivating controlled conversation construction.The paper identifies opening lines and subsequent interaction design as key quality factors.
- UltraChat organizes instructions into Questions about the World, Creation and Generation, and Assistance on Existing Materials.
- The schema targets the breadth of human-AI interactions rather than narrow tasks such as question-answering, rewriting, or summarization.
- Information Access: Questions about the World provides broad information access across diverse topics.
- Conditional Information Creation: Creation and Generation covers producing new information under human-input conditions, including emails, stories, and plays.
- Information Transformation: Assistance on Existing Materials transforms user-provided information through rewriting, continuation, summarization, or inference.
- Together, the three-part design aims to comprehensively represent possible human-AI interactions and meaningful collaboration.
4 Data Construction
UltraChat is constructed to maximize diversity across concepts, writing tasks, and assistance on existing materials, while preserving coherent multi-turn interaction. Its analysis reports strong scale, length, diversity, and coherence relative to other instruction datasets, and UltraLLaMA is trained on it with context-aware loss masking.
- Data sectors: UltraChat organizes data into three sectors: real-world concepts and entities, text generation, and assistance on existing materials.The sectors target breadth across topics, writing instructions, and transformations such as rewriting, translation, summarization, and question answering.
- Concepts and entities: The first sector expands 30 broad topics into 30–50 subtopics, then generates multiple question variants for each concept.It also incorporates frequently occurring real-world entities derived from Wikidata and Wikipedia.
- Text generation: The writing sector covers 20 material types and recursively elaborates generated instructions before using them as dialogue openings.Approximately 80% of the generated instructions are fed back to ChatGPT for more detailed instructions.
- Existing materials: The existing-materials sector combines 10,000 C4 text pieces with five instructions each, producing 500,000 concatenated dialogue openings.Keywords classify source URLs by material type, and manually designed templates combine texts with instructions.
- Dialogue generation: User-personality prompts and sector-specific reminders reduce role exchange and encourage more natural dialogue flow during automatic generation.A further filtration step is applied after generation.
- Data analysis: UltraChat has the largest scale among the compared publicly available datasets, the highest average turns and longest average instance length, while ranking highest in lexical diversity and coherence.Its average dialogue length is 1467.4 tokens; it slightly trails GPT4ALL in topic diversity but exceeds the other datasets.
- UltraLLaMA: UltraLLaMA is trained from LLaMA-13B on UltraChat using 2048-token dialogue sequences and response-only loss calculation to retain preceding conversational context.The training procedure breaks dialogues into shorter sequences and computes loss only on model responses.
5 Evaluation
The evaluation compares UltraLLaMA with open-source and other chat models using curated questions, ChatGPT-based pairwise judgments, independent scoring, and TruthfulQA. UltraLLaMA generally leads open-source baselines, while performance varies by task complexity and backbone model.
- Evaluation Setup: The curated evaluation set combines the Vicuna benchmark with 300 GPT-4-generated questions spanning knowledge, reasoning, mathematics, writing, and varied difficulty levels.The set includes commonsense, world knowledge, physics, biology, response generation, and writing tasks.
- Evaluation Setup: ChatGPT scores paired model responses from 1 to 10, prioritizing correctness, while randomized response order addresses presentation-order effects.The evaluation also counts Win/Tie/Lose outcomes for each baseline comparison.
- Response Comparison: UltraLLaMA beats every open-source model in pairwise comparison, reaching an 85% winning rate and a 13% higher winning rate than Vicuna.The comparisons are conducted on the curated evaluation set and assessed by ChatGPT.
- Independent Scoring: Independent ChatGPT scoring gives UltraLLaMA the highest overall score and the top result on nearly every evaluation segment.Table 7 reports average scores on a 1-to-10 scale for overall and segment-level performance.
- Performance Analysis: Simpler commonsense and world-knowledge questions are easier for all models, whereas reasoning and creative-writing tasks challenge most systems; backbone choice also matters.Alpaca performs relatively well on simpler questions despite having 7 billion parameters, while Dolly and OpenAssistant underperform similarly sized or smaller LLaMA-based models.
- TruthfulQA Results: TruthfulQA remains difficult: the best model only slightly exceeds 50% accuracy, while UltraLLaMA performs comparably to Vicuna and surpasses the other baselines.The reported task evaluates whether answers to TruthfulQA multiple-choice items are true or false.
6 Conclusion
The paper introduces UltraChat as a structured, broad multi-turn instructional dataset and uses it to create UltraLLaMA, which outperforms prior open-source models across multiple evaluations.
- UltraChat structures multi-turn instructional conversations across diverse topics and instructions to support general chat models.
- Fine-tuning LLaMA on UltraChat produced the conversational model UltraLLaMA.
- UltraChat shows strong scale, average length, diversity, and consistency, while UltraLLaMA surpasses open-source models including Vicuna, Alpaca, and Koala.
Limitations
The authors identify evaluation bias and limited evaluation coverage, restrict UltraChat to English, and note broader risks and resource costs associated with UltraLLaMA.
- Evaluating large language model response quality is challenging, and assessments may contain biases.
- The evaluation does not yet comprehensively test reasoning and multi-turn dialogue, and it uses ChatGPT rather than the more reliable GPT-4.
- UltraChat currently supports only English, although the authors are working toward adding languages such as Chinese.
- UltraLLaMA may face hallucination and misuse-related ethical problems, while its training is more energy-intensive than that of lightweight models.
A Prompts
This section provides the prompts used for evaluation.
- The section contains prompts used for evaluation.
- The prompts support the paper’s evaluation procedure.
- Evaluation prompt materials are presented in this section.
Comparison Evaluation Prompt
The comparison prompt asks an evaluator to assess two assistants’ responses for quality, helpfulness, detail, and harmlessness, then assign separate overall scores and explain the evaluation.
- The assistant-answer boundaries are explicitly marked for both Assistant 1 and Assistant 2.
- The evaluator compares two AI assistants’ responses to the displayed user question.
- The prompt asks for ratings of quality, helpfulness, detail, and harmlessness, summarized by an overall 1-to-10 score for each assistant.
- The evaluator must first output the two scores on one line, then provide a comprehensive explanation.
Independent Scoring Prompt
The prompt defines an AI-assistant response evaluation task that requests an overall integer score from 1 to 10. It emphasizes helpfulness, harmlessness, precision, quality, and detail.
- The assistant is instructed to be helpful, harmless, and precise when checking answer quality.
- The evaluation asks for feedback on response quality, helpfulness, detail, and harmlessness.
- The overall performance score ranges from 1 to 10, with higher values indicating better performance.
- The required output format is Score: followed by an integer between 1 and 10.
B Examples
The section presents examples of UltraLLaMa and compares two dialogue-generation approaches using the same opening line. The comparison contrasts direct ChatGPT querying with iterative User-AI simulations.
- Table 20 compares two dialogues that share the same opening line.
- The two dialogues are generated either by directly querying ChatGPT or by iteratively simulating User-AI interactions.