Source-linked AI summary
Yi: Open Foundation Models by 01.AI
01. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yanpeng Li, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, Zonghong Dai
TL;DR
The Yi family addresses how to build capable, deployable language and multimodal models through data engineering and extensions to context, vision-language, and depth. Using 6B and 34B models, the authors report GPT-3.5-matching performance and cost-effective deployment, while finding that further scaling with optimized data may strengthen future models.
Problem
The paper examines how data quality, model scale, and staged extensions can support strong capabilities in language and multimodal models.
Method
The authors pretrain standard dense transformers on heavily engineered corpora, then extend them through chat finetuning, long-context continual pretraining, vision-language adaptation, and depth upscaling.
Results
Yi-34B generally matches GPT-3.5 on standard benchmarks and is deployable on consumer-grade devices after 4/8-bit quantization.
Takeaways & Limitations
The results support pursuing large, thoroughly optimized datasets and continued parameter scaling as a route toward stronger frontier models and local deployment.
Takeaways & Limitations
The approach assumes that high-quality extensive data can make a standard dense transformer sufficiently capable, while benchmark evaluations are not used to optimize chat performance.
Abstract
from arXiv · showhide
We introduce the Yi model family, a series of language and multimodal models that demonstrate strong multi-dimensional capabilities. The Yi model family is based on 6B and 34B pretrained language models, then we extend them to chat models, 200K long context models, depth-upscaled models, and vision-language models. Our base models achieve strong performance on a wide range of benchmarks like MMLU, and our finetuned chat models deliver strong human preference rate on major evaluation platforms like AlpacaEval and Chatbot Arena. Building upon our scalable super-computing infrastructure and the classical transformer architecture, we attribute the performance of Yi models primarily to its data quality resulting from our data-engineering efforts. For pretraining, we construct 3.1 trillion tokens of English and Chinese corpora using a cascaded data deduplication and quality filtering pipeline. For finetuning, we polish a small scale (less than 10K) instruction dataset over multiple iterations such that every single instance has been verified directly by our machine learning engineers. For vision-language, we combine the chat language model with a vision transformer encoder and train the model to align visual representations to the semantic space of the language model. We further extend the context length to 200K through lightweight continual pretraining and demonstrate strong needle-in-a-haystack retrieval performance. We show that extending the depth of the pretrained checkpoint through continual pretraining further improves performance. We believe that given our current results, continuing to scale up model parameters using thoroughly optimized data will lead to even stronger frontier models.
1 Introduction
Yi presents 6B and 34B language models built around heavily engineered data, careful alignment, and extensions for context, vision-language, and depth scaling. The authors report GPT-3.5-matching performance, efficiency, and deployment-oriented benefits.
- Model Series: Yi introduces 6B and 34B language models pretrained from scratch on 3.1T tokens and finetuned with a small, meticulously polished alignment dataset.The models target a balance between inference feasibility on consumer hardware and sufficient scale for complex reasoning and emergent abilities.
- Data Engineering: The pretraining pipeline combines language, heuristic, perplexity, semantic, topic, and safety filtering with cascaded paragraph, MinHash, and exact-match deduplication.The authors attribute Yi’s performance primarily to data quality produced by this engineering effort.
- Capability Extensions: Yi extends model capabilities through 200K context scaling, vision-language adaptation, and depth-upscaling via continual pretraining.The paper reports that depth-upscaling further improves model performance.
- Results: Yi-34B generally matches GPT-3.5 on standard benchmarks and supports cost-effective deployment after parameter and KV-cache quantization.The comparison spans commonsense reasoning, college exams, mathematics, coding, reading comprehension, and human-preference evaluations.
- Impact: The model series is presented as enabling cost-effective research and applications, locally runnable chatbots, and further data-and-model scaling.The authors connect local deployment with user-data privacy and describe the models as usable for research and commercial purposes.
2 Pretraining
Yi’s pretraining uses a standard dense Transformer architecture trained on a large bilingual corpus whose quality is controlled through extensive filtering, clustering, deduplication, and topic-based sampling. The design also incorporates efficiency-oriented attention and activation choices and supports long contexts through RoPE adjustment and continual pretraining.
- Pretraining Strategy: Yi trains a standard dense Transformer on heavily engineered English-and-Chinese data, relying on data quality rather than major architectural modification.The data mixture contains 3.1T high-quality tokens from varied sources.
- Data Processing: The data pipeline begins with Common Crawl documents, applies language identification and perplexity scoring, then performs filtering and deduplication.Filtering targets low-quality text, while clustering supports quality labeling and mixture optimization.
- Model Architecture: Yi uses a modified decoder-only Transformer based on LLaMA, with Grouped-Query Attention in both Yi-6B and Yi-34B.The authors state that GQA reduces training and inference costs without observed performance degradation in Yi-6B.
- Model Architecture: SwiGLU reduces the activation size from 4h to 8/3h, compensating for parameter reductions associated with GQA.Here h denotes the hidden size.
- Long Context: RoPE with an adjusted base frequency supports context windows up to 200K, using continued pretraining and lightweight finetuning for long-context retrieval.The base model is trained at 4K context length, while the extended model uses long-sequence data and converges with 1–2B tokens according to the authors.
3 Finetuning
Yi’s finetuning prioritizes carefully examined, diverse data over dataset scale. The process uses iterative polishing, feedback, capability balancing, structured formatting, and response-focused optimization.
- Data Quality: The finetuning dataset contains fewer than 10K multi-turn instruction-response pairs, selected and polished through repeated examination and user feedback.The authors contrast this quality-focused approach with data-intensive datasets containing millions of entries.
- Response Quality: The finetuning process explicitly reduces hallucination and repetition by screening knowledge content and rewriting repetitive response turns.These interventions are applied during dataset construction and polishing.
- Data Diversity: Instruction diversity covers question answering, creative writing, dialogue, reasoning, mathematics, coding, safety, and bilingual capabilities.An instruction-tagging and diversity-focused sampling system balances the distribution across capability directions.
- Data Mixture: An approximate grid search determines capability-mixture ratios using validation results and in-house human evaluation sets.The search tests proportions from 1 to 1/64 for each ability.
- Training Format: Yi uses ChatML formatting to distinguish system configurations, user inputs, and assistant responses.The training loss is computed only on responses rather than system and user instructions.
4 Infrastructure
Yi’s infrastructure supports large-scale pretraining, finetuning, serving, and long-context training through elastic scheduling, reliability mechanisms, parallelism, multi-model orchestration, and inference optimization. These systems target both training efficiency and affordable deployment.
- Full-Stack Infrastructure: The infrastructure manages full-stack data processing, pretraining, finetuning, and serving with automated resource management and optimized distributed strategies.It supports heterogeneous distributed backends and serving accelerations.
- Resource Management: Multi-cloud scheduling and elastic scaling assign training jobs across available GPU nodes while scaling related hyperparameters seamlessly.The system is designed for long-running pretraining, SFT, and RLHF workloads with different priorities.
- Training Efficiency: Training efficiency addresses memory and communication limits through ZeRO-1, tensor and pipeline parallelism, kernel fusion, and topology-aware allocation.These techniques reduce optimizer-state memory, inter-node communication, redundant memory access, and cross-switch traffic.
- Finetuning Infrastructure: The finetuning framework orchestrates multiple models and caches reference-model outputs in DPO workflows to approach supervised-finetuning speed and cost.The framework supports different distributed backends within a single job.
- Inference: Quantization, dynamic batching, and Paged Attention reduce inference memory use and decoding cost, with near-zero performance degradation reported for quantized models.The cited implementation includes 4-bit model quantization and 8-bit KV-cache quantization.
- Long-Context Infrastructure: Engineering changes support 200K-context continued pretraining and finetuning while retaining full attention rather than using sparse or local attention.The implementation uses computation-communication overlap, sequence parallelism, and communication compression.
5 Safety
Yi develops a full-stack Responsible AI Safety Engine covering safety during pretraining, alignment, and deployment. Its pretraining and alignment measures target privacy, harmful content, and a broad taxonomy of safety concerns.
- RAISE provides full-stack safety coverage across pretraining, alignment, and deployment.The paper presents RAISE as the framework for enhancing model trustworthiness and safety.
- Safety in Pretraining: Pretraining filters remove personal identifiers and private data while reducing sexual, violent, and extremist content.The filters combine heuristic rules, keyword matching, and learned classifiers.
- Safety in Alignment: Alignment uses a comprehensive taxonomy spanning privacy, discrimination, violence, self-harm, sexual content, mental health, and cybersecurity threats.The taxonomy also covers environmental, religious, substance-abuse, illegal-activity, hate-speech, and ethical concerns.
6 Evaluations
Yi models are evaluated across academic benchmarks, scaling comparisons, in-context learning, and chat assessments. The results show strong overall performance, benefits from model scale and data quality, near GPT-3.5-level chat performance, and remaining gaps on several reasoning tasks.
- 6.1.1 Main Results: 3.1T training tokens produced substantial performance gains across benchmarks, although mathematics and coding remained weaker areas.The authors note that continual pretraining and instruction fine-tuning could improve these domains.
- 6.1.2 Discussions: Yi-34B substantially outperformed Yi-6B despite using the same pretraining corpora, with larger gains on coding and mathematics benchmarks.The comparison reports smaller differences on commonsense reasoning, reading comprehension, and knowledge tasks.
- 6.1.2 Discussions: Smaller models trained on higher-quality data, including Yi-34B and Qwen-14B, generally outperformed larger models associated with lower-quality data, such as Falcon-180B.The paper presents this comparison as evidence concerning data quality and model performance.
- 6.1.2 Discussions: Open-source models still lagged GPT-4 and GPT-3.5 overall, while Yi-34B could match or surpass GPT-4 on Chinese knowledge benchmarks but remained behind on reasoning, coding, and mathematics.The cited Chinese knowledge benchmarks are C-Eval, CMMLU, and Gaokao; the reasoning-related examples are BBH, HumanEval, and MATH.
- 6.2.2 Human Evaluations: Yi data showed a clear tendency toward preference improvements as the dataset scaled, compared with UltraChat and UltraChat 200K.The authors use the speed of preference increase during data scaling to demonstrate data quality.
7 Capability Extension
Yi extends its base models to 200K context, vision-language, and deeper architectures through lightweight continual pretraining, finetuning, and multimodal alignment. These extensions preserve short-context capability, support strong long-context retrieval, and yield competitive depth-upscaled results.
- Overview: Yi extends its capabilities through 200K context scaling, vision-language adaptation, and depth-upscaling.The long-context and depth extensions use continual pretraining, while vision-language adaptation integrates a vision encoder with the language model.
- 7.1 Long-Context: Lightweight continual pretraining and finetuning adapt Yi to 200K contexts while retaining short-context generic capability.The approach uses long-context data and document question-answering data; Table 6 reports no significant change in short-context capability after adaptation.
- 7.1 Long-Context: Figure 6 evaluates document length against needle depth and shows near-all-green Needle-in-a-Haystack performance for Yi-34B-200K.The authors nevertheless characterize this retrieval level as relatively easy for long-context language models.
- 7.2 Vision-Language: Yi-VL combines a CLIP ViT-H/14 image encoder, a two-layer projection module, and Yi chat language models through staged training.The vision encoder produces image representations, while the projection module aligns them with the language model’s text feature space.
- 7.3 Depth-Upscaling: Duplicating layers selected by input-output cosine similarity enables targeted depth scaling with minimal performance impact.The newly added layers show cosine similarity scores nearly equal to one, and Yi-9B remains competitive across common-sense, reasoning, knowledge, coding, and mathematics benchmarks.
8 Final Discussions
The final discussion presents Yi-34B as a GPT-3.5-matching model that can run on consumer-grade devices, and highlights data quantity, data quality, and iterative finetuning as central lessons. The authors also identify continued scaling with optimized data as a direction for stronger future models.
- Deployment: Yi-34B achieves GPT-3.5-matching performance and is deployable on consumer-grade devices through 4/8-bit quantization.The paper frames this combination as suitable for local deployment.
- Pretraining: 3.1T pretraining tokens and data-quality engineering are presented as key factors in Yi’s performance.The authors emphasize both data quantity beyond Chinchilla-optimal scaling and source selection, filtering, and deduplication quality.
- Finetuning: Small-scale finetuning data of ≤10K examples is iteratively polished case by case by machine learning engineers using real user feedback.The authors contrast this procedure with instruction-scaling approaches associated with FLAN and UltraChat.
- Future Scaling: The authors report that reasoning capability is strongly correlated with model scale when pretraining data is fixed and propose scaling parameters with optimized data.They present this as a direction for upcoming model versions and stronger frontier models.
A Author List and Contributions
The paper credits contributors across frontier research, infrastructure, pretraining, finetuning and alignment, and safety. Team members are listed alphabetically, with equal contribution stated.
- Contributions: The Yi development team contributed across frontier research, machine learning infrastructure, pretraining, finetuning and alignment, and safety.These contribution areas span research, engineering, training, alignment, and responsible AI work.
- Author List: The authors are listed in alphabetical order and all authors contributed equally to the work.The contribution statement applies to the complete author list.