PaperlayerView exact revision on arXiv

Source-linked AI summary

DISC-FinLLM: A Chinese Financial Large Language Model based on Multiple Experts Fine-tuning

Wei Chen, Qiushi Wang, Zefei Long, Xianyin Zhang, Zhongtian Lu, Bingxuan Li, Siyuan Wang, Jiarong Xu, Xiang Bai, Xuanjing Huang, Zhongyu Wei

arXiv:2310.15205v2cs.CL

TL;DR

Financial NLP models and general-domain LLMs have limitations in financial knowledge, multitasking, computation, and Chinese-market coverage. DISC-FinLLM uses a four-part instruction dataset and multiple expert fine-tuning; evaluations report improvements over baseline models across financial tasks and domains.

  • Problem

    Financial NLP models face gaps in specialized knowledge, annotated data, numerical computation, real-time response, and cross-task generalization, while general-domain LLMs lack comprehensive financial knowledge, especially for the Chinese market.

    Source: §1 · p. 1, §2.1 · p. 3

  • Method

    DISC-FinLLM combines DISC-FIN-SFT, covering consulting, financial NLP, computing, and retrieval, with four scenario-specific LoRA modules on a Baichuan-13B backbone.

    Source: §1 · pp. 1, 3

  • Results

    DISC-FinLLM outperforms its base foundation model across downstream tasks, with evaluations reporting effectiveness across financial NLP, human testing, computation, and retrieval domains.

    Source: §1 · p. 3, §6 · p. 10

  • Takeaways & Limitations

    The authors identify financial customer support, investment analysis, and risk assessment as possible applications of the model's performance across evaluated domains.

    Source: §6 · p. 10

    Financial NLP remains constrained by costly annotated data, limited numerical computation, and weak cross-task generalization.

    Source: §2.1 · p. 3

Abstract

from arXiv · show

We propose Multiple Experts Fine-tuning Framework to build a financial large language model (LLM), DISC-FinLLM. Our methodology improves general LLMs by endowing them with multi-turn question answering abilities, domain text processing capabilities, mathematical computation skills, and retrieval-enhanced generation capabilities. We build a financial instruction-tuning dataset named DISC-FIN-SFT, including instruction samples of four categories (consulting, NLP tasks, computing and retrieval-augmented generation). Evaluations conducted on multiple benchmarks demonstrate that our model performs better than baseline models in various financial scenarios. Further resources can be found at https://github.com/FudanDISC/DISC-FinLLM.

1 Introduction

Financial NLP and general-domain LLMs face gaps in financial knowledge, multitasking, numerical computation, and Chinese-market coverage. DISC-FinLLM combines a four-part instruction dataset with specialized fine-tuning, and outperforms its base foundation model across downstream tasks.

  • Financial NLP models face scarce labeled data, limited financial knowledge and multitasking, weak numerical computation, and inadequate real-time information handling.Source: §1 · p. 1
  • General-domain LLMs typically lack comprehensive financial knowledge, especially for the Chinese market, motivating an open-source Chinese financial LLM for varied users and scenarios.Source: §1 · p. 1
  • DISC-FIN-SFT covers consulting, financial NLP tasks, computing, and retrieval-enhanced instructions.The instructions draw on financial Q&A, existing and constructed NLP datasets, computational problems, and financial texts with generated questions, retrieved references, and answers.Source: §1 · pp. 1, 3
  • MEFF trains four LoRA modules for financial dialogue, NLP tasks, calculation, and retrieval question answering, using Baichuan-13B as the backbone.Source: §1 · p. 3
  • DISC-FinLLM outperforms its base foundation model across all downstream tasks on multiple evaluation benchmarks.Source: §1 · p. 3

2 Related Work

Prior financial NLP models have progressed across specialized tasks but face challenges in data, domain knowledge, numerical reasoning, real-time response, and generalization. Financial LLM work includes large pretrained models and data-centric adaptation, while training-data and application coverage remain concerns.

2.1 Financial NLP Models and Limitations

  • Financial NLP models have addressed tasks including sentiment analysis, event extraction, report generation, and summarization, but face costly annotation, weak specialized knowledge, limited inference, real-time, numerical, and cross-task capabilities.Source: §2.1 · p. 3

2.2 Large Language Models for Finance

  • BloombergGPT and XuanYuan 2.0 use targeted financial or Chinese-domain pretraining, while FinGPT adopts low-rank adaptation and PIXIU focuses on financial NLP datasets with human-written prompts.The passage notes that PIXIU's training-data scope limits its wider application, while FinGPT still requires downstream fine-tuning.Source: §2.2 · pp. 3–4

3 DISC-Fin-SFT Datasets

DISC-Fin-SFT is an approximately 250,000-example dataset covering financial consultation, NLP tasks, computing, and retrieval-enhanced generation. Its instructions combine existing datasets and generated material to support varied financial tasks.

3 DISC-Fin-SFT Datasets

  • Approximately 250,000 examples span financial consulting, task instructions, financial computing, and retrieval-enhanced instructions.The four categories form the dataset’s main components.Source: §3 · p. 4

3.1 Financial Consulting Instructs

  • Consulting instructions combine translated FiQA questions with regenerated answers, finance-term QA pairs, and forum-topic multi-turn dialogues.Prompts are designed to align responses with Chinese national conditions, stance, attitude, and language style.Source: §3.1 · pp. 4–5

3.2 Financial Task Instructions

  • Task instructions draw on over 10 Chinese financial NLP datasets and unlabeled financial texts expanded into 1.8 million paragraph-level units.The dataset-based instructions use human-written prompts; the unlabeled-text material is converted into reading-comprehension QA triplets.Source: §3.2 · p. 5

3.3 Financial Computing Instructions

  • Financial computing instructions use seed questions and plugin commands, then add over 50,000 generated question-answer pairs for tool-use training.Seed questions include financial exam calculations, report-based arithmetic, and general mathematics; the training is intended to teach appropriate use of four tools.Source: §3.3 · pp. 5–6

3.4 Retrieval-enhanced Instructions

  • Retrieval-enhanced instructions are built by generating financial questions, retrieving relevant references with a threshold, and generating suggestions from both.Questions and answers are generated through Chain-of-Retrieval prompting.Source: §3.4 · p. 6
  • The 20,000 retrieval-enhanced instructions cover industry analysis, policy analysis, investment guidance, and other financial scenarios.The instructions guide reference comprehension and summarization and help identify irrelevant documents; the reference base includes research-report abstracts and financial news.Source: §3.4 · pp. 6–7

4 Multiple Experts Fine-tuning Framework

DISC-FinLLM uses task-specific LoRA modules to separate financial capabilities. Four experts address consultation, document processing, computation, and knowledge retrieval, with modules switched according to task needs.

4.1 Overall Architecture

  • Four task-specific LoRA modules can be switched on the base model to activate or deactivate functions without retraining the entire model.The authors describe this arrangement as supporting efficiency and improved performance.Source: §4.1 · p. 7

4.2 Multiple Experts Setup

  • The four experts target financial consultation, document processing, computing, and knowledge retrieval, each trained on its corresponding instruction category.The framework trains individual components on specialized datasets so they can operate autonomously without interfering with one another.Source: §4.2 · p. 7, §4.1 · p. 7
  • The consultation expert handles financial Q&A, while document processing covers tasks such as information extraction and sentiment analysis.The consultation data is tailored to the Chinese financial context.Source: §4.2 · p. 7
  • The computing expert uses four tools for calculations, while the retrieval expert retrieves and analyzes documents on news, reports, and policies.Tool calls interrupt decoding and insert tool results into generated text; retrieval can support responses on current affairs, industry trends, and financial policies.Source: §4.2 · pp. 7–8

5 Experiments

Across financial NLP, human-question, calculation, and retrieval evaluations, DISC-FinLLM and its task-specific LoRA models show strong results against comparison models. The setup combines benchmark tests with manually constructed calculation and current-affairs retrieval evaluations.

5.1 Evaluation Setup

  • Evaluation covered six FinCUGE NLP tasks, FinEval multiple-choice questions, over 100 manually created calculation problems, and current-affairs retrieval judged on four GPT-3.5 metrics.The evaluation framework comprises financial NLP tasks, human tests, data analysis, and current-affairs analysis.Source: §5.1 · p. 8

5.2 Main Results

  • Average performance on six unseen financial NLP tasks was 2 to 9 points higher than that of untrained base models.Some evaluated tasks were not covered by the instruction data.Source: §5.2 · p. 8, Table 3 · p. 9
  • Four task-specific LoRA models achieved the best average human-test results except ChatGPT and GPT-4, while full-data fine-tuning decreased results.The authors also report superior performance to existing financial LLMs.Source: Table 4 · p. 9, §5.2 · p. 10
  • With computational plugins, the model surpassed ChatGPT by 0.09 points on financial computing tasks.Source: Table 5 · p. 9, §5.2 · p. 10
  • DISC-FinLLM showed significantly higher results on all four retrieval-test metrics: accuracy, usefulness, linguistic quality, and reflectiveness.GPT-3.5 evaluated the generated results.Source: Table 6 · p. 9, §5.2 · p. 10

6 Conclusion

The conclusion presents DISC-FinLLM as a Chinese financial system built with multiple-expert fine-tuning and task-specific instruction data. Its reported evaluations span four financial task domains, with potential applications in customer support, investment analysis, and risk assessment.

  • The framework uses task-specific LoRA instruction tuning and system prompts across financial NLP, human tests, computation, and retrieval tasks.Source: §6 · p. 10
  • The authors report effective performance across the four evaluated financial domains.Source: §6 · p. 10
  • The authors identify financial customer support, investment analysis, and risk assessment as possible applications of the model’s performance.Source: §6 · p. 10

A Prompt Engineering

The prompt templates construct financial instructions through question and answer generation, with different procedures for consulting, multi-turn dialogue, NLP tasks, computation, and retrieval-enhanced tasks. The templates include both zero-shot and few-shot approaches.

  • Consulting templates generate answers to questions with optional context and no demonstrations, or generate questions from financial terms and unlabeled text with few-shot examples.Source: A Prompt Engineering · p. 12, Figure 5 · p. 13, Figure 6 · p. 13
  • Multi-turn instructions are generated one dialogue turn at a time, using the topic and dialogue history for questions and context plus history for detailed answers.The iterative approach responds to generated answers being too short and lacking detail when applying the referenced method directly.Source: A Prompt Engineering · p. 14, Figure 8 · p. 14
  • Non-generative financial NLP tasks use zero-shot and few-shot templates, while computation prompts use few-shot chain-of-thought to invoke tools.Source: A Prompt Engineering · p. 14, Figure 9 · p. 14
  • Retrieval-enhanced templates generate questions from financial news and research reports, then generate answers from retrieved documents and the question.Source: A Prompt Engineering · p. 14

B Details of Financial NLP Task Instructions

The financial NLP instruction datasets cover information extraction, sentiment analysis, text classification, text generation, and question answering. They draw on multiple existing datasets and include translated or generated material.

Information Extraction

  • Information-extraction datasets support structured information construction from financial text, including entity recognition, event selection, causal relations, and intent detection.The described sources include Financial Report, SmoothNLP, CCKS, OpenKG, and minds14.Source: B Details of Financial NLP Task… · pp. 14, 16

Sentiment Analysis

  • Sentiment instructions use FPB, FiQA-SA, and FNSC; the two English datasets were translated into Chinese, and FiQA-SA includes discrete labels and scores from -1 to 1.Source: B Details of Financial NLP Task… · p. 16

Text Classification

Text Generation

  • Text-generation instructions cover financial-news headline generation with SmoothNLP and keyword generation from Wealth-alpaca questions using ChatGPT-generated terms.The Wealth-alpaca questions are paired with 2–3 key terms as instruction outputs.Source: B Details of Financial NLP Task… · pp. 16–17

Question Answering

C Details of FinCUGE evaluation benchmark

FinCUGE evaluates financial NLP across six datasets that do not overlap with DISC-FinLLM-SFT data, although some task types recur. The limited number of high-quality Chinese financial NLP datasets constrains evaluation on completely unseen task types.

C Details of FinCUGE evaluation benchmark

  • FinCUGE’s six evaluation datasets are strictly nonoverlapping with DISC-FinLLM-SFT data, but some task types are shared.The paper notes that the relatively small supply of high-quality Chinese financial NLP datasets hinders testing zero-shot generalization on NLP tasks.Source: C Details of FinCUGE evaluation… · p. 17
  • FinFE tests negative-neutral-positive sentiment classification of financial social media text, measured by accuracy.Its train, validation, and test sets contain 8,000, 1,000, and 1,000 articles, respectively.Source: C Details of FinCUGE evaluation… · p. 17
  • FinQA tests question answering about events in financial news or announcements, measured by F1-Score.It is derived from DuEE-fin; its train, validation, and test sets contain 16,000, 2,000, and 2,000 articles.Source: C Details of FinCUGE evaluation… · p. 17
  • FinCQA tests answers to questions about causal events in financial news, measured by F1-Score.Its train, validation, and test sets contain 21,965, 2,741, and 2,745 articles, respectively.Source: C Details of FinCUGE evaluation… · p. 17
  • FinNA tests financial news summarization, measured by Rouge.Its train, validation, and test sets contain 24,000, 3,000, and 3,000 articles, respectively.Source: C Details of FinCUGE evaluation… · p. 17
  • FinRE tests classification of relations between head-tail entity pairs in financial news, with up to 44 categories including null, measured by F1-Score.Its train, validation, and test sets contain 7,454, 1,489, and 3,727 articles, respectively.Source: C Details of FinCUGE evaluation… · p. 17
  • FinESE tests extraction of subjects for a specified event type from financial news, measured by F1-Score.Its train, validation, and test sets contain 14,252, 1,781, and 1,782 articles, respectively.Source: C Details of FinCUGE evaluation… · p. 17
Loading 2310.15205v2…