Source-linked AI summary
XuanYuan 2.0: A Large Chinese Financial Chat Model with Hundreds of Billions Parameters
Xuanyu Zhang, Qing Yang, Dongliang Xu
TL;DR
Open-source Chinese chat models at the hundreds-of-billions scale, especially for finance, were lacking. The paper introduces XuanYuan 2.0 on BLOOM-176B and hybrid-tuning, reporting accurate, contextually appropriate Chinese financial responses while combining general and financial knowledge.
Problem
Open-source chat models at the hundreds-of-billions scale were unavailable for Chinese, particularly the Chinese financial domain.
Method
XuanYuan 2.0 combines BLOOM-176B’s decoder-only architecture with hybrid-tuning that integrates general and financial data across pre-training and instruction fine-tuning.
Results
XuanYuan 2.0 delivers precise and contextually relevant responses in the Chinese financial domain while leveraging general and domain-specific knowledge.
Takeaways & Limitations
The model provides an open-source large-scale Chinese financial chat system designed to retain general conversational capabilities alongside financial-domain competence.
Abstract
from arXiv · showhide
In recent years, pre-trained language models have undergone rapid development with the emergence of large-scale models. However, there is a lack of open-sourced chat models specifically designed for the Chinese language, especially in the field of Chinese finance, at the scale of hundreds of billions. To address this gap, we introduce XuanYuan 2.0, the largest Chinese chat model to date, built upon the BLOOM-176B architecture. Additionally, we propose a novel training method called hybrid-tuning to mitigate catastrophic forgetting. By combining general-domain with domain-specific knowledge and integrating the stages of pre-training and fine-tuning, XuanYuan 2.0 is capable of providing accurate and contextually appropriate responses in the Chinese financial domain.
1 Introduction
Large language models have advanced rapidly, but Chinese financial NLP still lacks open-source models at the hundreds-of-billions scale. XuanYuan 2.0 addresses this gap with a large Chinese financial chat model and hybrid-tuning to retain general capabilities while using financial knowledge.
- Motivation: Domain-specific models must capture field-specific language, terminology, and context, but training only on domain-specific data can cause catastrophic forgetting.The introduction motivates combining general and domain-specific training signals.
- Motivation: Existing Chinese financial models such as FinBERT, Mengzi, and FinT5 have fewer than one billion parameters, limiting their ability to meet growing financial NLP demands.These models target financial text analysis and understanding, but the paper identifies a need for more powerful systems.
- Contribution: XuanYuan 2.0 is an open-source Chinese chat model built on BLOOM-176B for the Chinese financial domain.The paper presents it as the largest Chinese chat model to date and as a response to the absence of comparable open-source models.
- Contribution: Hybrid-tuning combines general-domain and financial knowledge while integrating pre-training and instruction fine-tuning to mitigate catastrophic forgetting.The method is intended to preserve general generation capabilities during financial instruction tuning.
2 Related Work
Prior work established general-purpose and domain-specific pretrained models, including financial systems, but large-scale open-source chat models for Chinese finance remained limited. XuanYuan 2.0 is proposed to address that unresolved need.
- General Pretraining: General pretrained architectures include encoder, decoder, and encoder-decoder models represented by BERT, GPT, and T5.These architectures serve different NLP requirements.
- Domain-Specific Models: Domain-specific pretraining incorporates specialized knowledge to capture linguistic nuances and improve performance on tasks within a target domain.Examples discussed include biomedical models and BloombergGPT for finance.
- Research Gap: Large-scale open-source chat models tailored specifically to Chinese and Chinese finance remained limited, motivating XuanYuan 2.0.The model is built upon BLOOM-176B with hundreds of billions of parameters.
3 XuanYuan 2.0
XuanYuan 2.0 uses BLOOM’s decoder-only autoregressive architecture and a hybrid-tuning framework that jointly trains general and financial data. The system combines this training design with large-scale data and distributed infrastructure for Chinese financial conversation.
- Model Architecture: XuanYuan 2.0 adopts BLOOM’s decoder-only Transformer architecture with autoregressive next-token prediction.The model also uses ALiBi positional embeddings and embedding LayerNorm.
- Hybrid-tuning: Hybrid-tuning integrates pre-training and instruction fine-tuning data from both general and financial domains into one training process.The approach randomly shuffles general pre-training, financial pre-training, general instruction, and financial instruction data.
- Hybrid-tuning: The one-stage hybrid-tuning process is designed to handle financial instructions while retaining general conversational capabilities.This design addresses catastrophic forgetting in domain-specific training.
- Training Data: Instruction data combines general examples collected with Self-Instruct and financial examples gathered through Self-QA from structured and unstructured financial sources.Unsupervised pretraining data is crawled, cleaned, and filtered from the Internet.
- Training Infrastructure: Training uses NVIDIA A100 80GB GPUs, DeepSpeed, pipeline parallelism, and ZeRO stage 1.Pipeline parallelism distributes model layers across GPUs, while ZeRO stage 1 partitions optimizer states.
4 Experiment
The experiment compares XuanYuan with open-source Chinese conversational models using manually assessed general- and financial-domain evaluation datasets. The reported findings indicate strong financial-domain knowledge and conversational capabilities, while further evaluation details were deferred.
- Evaluation: The study compares XuanYuan with other open-source Chinese conversational models.The comparison is conducted alongside purpose-built evaluations covering general and financial domains.
- Evaluation: The evaluation datasets cover multiple dimensions across both general and financial domains and are manually assessed.This setup is intended to examine conversational performance in both broad and financial settings.
- Results: The reported results show robust knowledge and conversational capabilities for XuanYuan in the financial domain.Additional findings were planned for a later version after evaluation rankings were released.
5 Conclusion
XuanYuan 2.0 addresses the lack of open-source, billion-scale Chinese financial chat models with a hybrid-tuning approach designed to preserve general and domain-specific knowledge.
- XuanYuan 2.0 fills the gap in open-source billion-scale chat models designed specifically for the Chinese financial domain.
- Hybrid-tuning combines general-domain and domain-specific knowledge while integrating pre-training and fine-tuning to mitigate catastrophic forgetting.
- The model delivers precise and contextually relevant responses within the Chinese financial domain.