FinGPT: Instruction Tuning Benchmark for Open-Source Large Language Models in Financial Datasets

• The paper identifies integration with diverse financial datasets, performance on financial terminology, and adaptability across tasks as challenges for existing approaches.Source: §1 · p. 2
• The proposed scheme uses Instruction Tuning to integrate open-source LLMs with financial data, emphasizing transparency, reproducibility, and plug-and-play integration.Source: §1 · p. 2
• The approach is tailored to improve transformer models’ adaptability and relevance across financial datasets while addressing integration challenges.Source: §1 · p. 2
• The cost-effective, end-to-end benchmark evaluates financial LLMs from basic competencies through complex multi-task operations.Source: §1 · p. 2

2 Related Works

• Existing financial GPT research generally uses prompt engineering with unchanged parameters or supervised fine-tuning such as Instruction Tuning for domain-centric models.Source: §2 · p. 2

2.1 General Large Language Models

• The surveyed general models include Llama2, ChatGLM2, BLOOM, Falcon, MPT, and Qwen, with descriptions of their language support and model characteristics.Source: §2.1 · pp. 2–3

2.2 Financial Large Language Models

• Financial model work includes FinBert, FLUE/FLANG-BERT, BloombergGPT, FinGPT, and PIXIU, spanning specialized models, benchmarks, and open- or closed-source systems.Source: §2.2 · p. 3
• Financial task evaluations mainly use Llama models, leaving the performance of other open-source models less understood across tasks.Source: §2.2 · p. 3
• The paper proposes a more integrated paradigm for connecting open-source LLMs with intricate financial data.Source: §2.2 · p. 3

3 Proposed Paradigm

• The paradigm begins with ongoing financial-task evaluation and instruction construction, then selects and evaluates a base model.Source: §3 · p. 4
• Its three phases are designed to facilitate integration and analysis across financial NLP datasets.Source: §3 · p. 4
• The paradigm is presented as an interconnected process rather than an isolated task evaluation.Source: §3 · p. 4

3.1 Task-Specific Instruction Tuning

• The task-specific phase evaluates financial LLMs on isolated NLP tasks to assess their capabilities and identify areas for improvement.It examines their ability to extract, process, and analyze information from financial data sources.Source: §3.1 · p. 4

3.1.1 Potential Challenges in Task-Specific Instruction Tuning

• Financial NLP tasks vary in complexity and specificity, so models may struggle with tasks requiring deeper financial understanding or advanced analytical skills.Source: §3.1.1 · p. 4
• Dataset quality and reliability affect tuning effectiveness, making accuracy, relevance, completeness, and representative sampling important.Source: §3.1.1 · p. 4
• Choosing metrics and benchmarks that accurately compare models across distinct financial tasks is challenging.Source: §3.1.1 · p. 4

3.1.2 Addressing the Challenges in Task-Specific Instruction Tuning

• The proposed response adapts tuning depth and breadth to each task’s complexity.Source: §3.1.2 · p. 4
• The approach validates financial datasets for accuracy, completeness, and relevance, and develops task-appropriate metrics and benchmarks for ongoing refinement.Source: §3.1.2 · p. 4

3.2 Multi-Task Instruction Tuning

• Multi-task instruction tuning integrates several financial NLP tasks to evaluate LLM versatility and adaptability across concurrent operations.The section presents this as relevant to practical utility and efficiency in financial scenarios.Source: §3.2 · pp. 4–5

3.2.1 Potential Challenges in Multi-Task Instruction Tuning

• Concurrent training can cause task interference, increase computational complexity, and make balanced task weighting difficult.Source: §3.2.1 · p. 5

3.2.2 Addressing Challenges in Multi-Task Instruction Tuning

• The proposed response combines optimized computation, efficient training protocols, and advanced evaluation techniques to address multi-task challenges.Source: §3.2.2 · p. 5
• Efficient batching and parallel processing distribute computational loads to support training speed without sacrificing model quality.Source: §3.2.2 · p. 5
• Dynamic task weighting adjusts each task’s loss weight to promote balanced learning without allowing one task to dominate.Source: §3.2.2 · p. 5
• The authors state that these measures collectively address multi-task challenges and enable deployment of proficient LLMs in finance.Source: §3.2.2 · p. 5

3.3 Instruction Tuning for Zero-shot Ability

• The zero-shot phase tests LLM adaptability, learning agility, and responses to novel financial tasks using unseen tasks and novel datasets.The stated evaluation focuses on robustness, flexibility, and problem-solving abilities.Source: §3.3 · p. 5

3.3.1 Potential Challenges in Instruction Tuning for Zero-shot Ability

• The section introduces strategic approaches to navigate the challenges associated with evaluating novel tasks.Source: §3.3.1 · p. 5, §3.3.2 · p. 5
• Unfamiliar tasks can prompt plausible but unfounded responses, while balancing generalization to new tasks against specialization in learned tasks remains challenging.Source: §3.3.1 · p. 5

3.3.2 Addressing Challenges in Instruction Tuning for Zero-shot Ability

• Task reformulation gives models a more focused objective for categorizing inputs without generating extraneous information.The authors say this narrows the task gap between reformulated tasks and the target task.Source: §3.3.2 · p. 6
• Adaptive learning and fine-tuning dynamically adjust the learning process according to the task to balance generalization and specialization.Source: §3.3.2 · p. 6
• The authors state that their strategies mitigate identified challenges and promote zero-shot tuning for financial-domain LLMs.Source: §3.3.2 · p. 6

4.1 Data Preparation

• The implementation draws on financial datasets selected to cover sentiment analysis, headline classification, named entity recognition, and relation extraction.The selection is described as complementary and intended to support broad task coverage.Source: §4.1 · p. 6

4.2 Instruction Construction

• Instructions use a standardized prompt, input, and answer template, with task-specific instructions and classification reformulations for multi-task and zero-shot experiments.Zero-shot instructions add options; ChatGPT generated ten instructions per task with randomized option order in training.Source: §4.2 · pp. 6–7
• Table 1 describes tasks and datasets; its (CLS) labels denote reformulations into unified classification tasks, including entity-type and relation-type classification.Source: Table 1 · p. 7

4.3 Training Detail

• Six comparable-size open-source base models were instruction-tuned using LoRA; task-specific runs used 8 epochs for SA, HC, and RE and 50 for NER, and multi-task runs used 4 epochs.Across six base models and four tasks, task-specific jobs took 30 hours; multi-task and zero-shot jobs took about 60 hours, for a total training cost of $302.4.Source: §4.3 · pp. 7–8

5.1 Task-Specific Instruction Tuning

• Under task-specific tuning, Llama2 ranked second on average across tasks; BLOOM excelled at information extraction, while MPT led sentiment analysis.Falcon and Qwen showed balanced performance, while BLOOM recorded the lowest performance in sentiment analysis and health classification.Source: §5.1 · p. 8

5.2 Multi-Task Instruction Tuning

• Multi-task tuning caused minor declines for most models in sentiment analysis and health classification, while Qwen improved across all sentiment sub-datasets.BLOOM had the most substantial classification degradation after already limited task-specific performance.Source: §5.2 · p. 8
• All models significantly improved in relation extraction under multi-task tuning; Llama2 and MPT also progressed in NER, while BLOOM outperformed Falcon in information extraction.The authors suggest RE(CLS) and additional financial NLP tasks may have addressed potential under-fitting in task-specific tuning.Source: §5.2 · pp. 8–9
• MPT made the largest NER and RE gains despite weak task-specific performance, while Falcon and ChatGLM2 made moderate gains in both tasks.Source: §5.2 · p. 9

5.3 Instruction Tuning for Zero-shot Ability

• For zero-shot evaluation, the authors reformulated NER and RE as classification tasks and excluded neutral sentiment samples; ChatGLM2 and Falcon demonstrated generalization to unseen instructions.Original NER and RE tasks did not sufficiently activate zero-shot abilities because of insufficient instruction diversity, with hallucination and inconsistent option responses.Source: §5.3 · p. 10
• ChatGLM2 stood out in zero-shot sentiment analysis, Falcon followed closely, and both BLOOM and Qwen struggled, often misclassifying responses as positive.The reported zero-shot F1-scores cover sentiment-analysis test datasets for all base models.Source: §5.3 · p. 10, Table 5 · p. 10

• The study presents task-specific, multi-task, and zero-shot instruction tuning and reports distinct LLM performance patterns across financial NLP tasks.It frames these analyses as insights into employing models for specific financial applications.Source: §6 · p. 10
• Future work will add open-source base models, examine 13–100-billion-parameter models, improve robustness and generalization, and address task interference and hallucination.The stated aim includes accurate and reliable responses across diverse tasks.Source: §6 · p. 10