InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning

• The authors identify inaccessible model parameters and weak generalization beyond instructed tasks as obstacles to financial LLM research.Source: §1 · p. 1
• InvestLM is tuned on LLaMA-65B with curated finance instructions and released with its model parameters; experts rate its responses comparable to commercial models.Source: §1 · p. 1
• Case studies show InvestLM mitigating hallucinations in LLaMA’s responses, and it often gives concise conclusions similar to commercial models.When commercial models give decisive answers, InvestLM typically reaches a similar logical investment conclusion.Source: §1 · p. 3
• InvestLM generalizes to several financial NLP tasks that were not used in its instruction tuning.Source: §1 · p. 3
• Diverse domain instructions effectively transform a strong foundation model, while generic instructions can hurt performance on domain tasks.The authors relate the first finding to the Superficial Alignment Hypothesis (Zhou et al., 2023).Source: §1 · p. 3

• The dataset reflects the premise that a pretrained model may already understand financial content, while specifically crafted instructions help it interact with investment questions.Source: §2 · p. 3
• Sources include high-vote quantitative finance discussions, CFA questions with explanations, finance journal articles, and textbook exercises with standard answers.Source: §2 · p. 3
• The dataset also includes questions answered from SEC filings and reformatted financial NLP tasks with manually written responses beyond the original labels.Source: §2 · p. 3
• For brainstormed investment questions, ChatGPT and Claude-2 generate candidate answers that are manually verified and paraphrased.Source: §2 · p. 3
• Each resource contributes several hundred instruction-input-output examples, and the dataset has longer inputs and responses than general-purpose instruction sets such as Alpaca.Table 2 reports average token lengths for instruction/input and response.Source: §2 · p. 3, Table 2 · p. 4
• The manually curated dataset covers financial investment topics using examples from several financial resources.Source: §2 · p. 4

• LoRA tunes LLaMA-65B for training efficiency, using rank 16 and six targeted projection modules.The targeted modules are q_proj, k_proj, v_proj, down_proj, gate_proj, and up_proj.Source: §3 · p. 4
• Linear Rope Scaling at scale 4 increases the context size to 8,192 to improve handling of long financial texts.Source: §3 · p. 4
• LLaMA-65B is trained for 15 epochs at a 3e-4 learning rate and batch size 16; the subsequent LLaMA-7B analysis uses 12 epochs, 3e-3, and batch size 32.The paper denotes InvestLM-65B as InvestLM and LLaMA-65B as LLaMA.Source: §3 · p. 4

4 Expert Evaluation

• Six financial experts compared responses to 30 manually written questions from InvestLM and GPT-3.5, GPT-4, and Claude-2; GPT-4 also evaluated responses using the same protocol.The commercial-model responses were sampled throughout August 2023.Source: §4 · p. 4
• Experts rated InvestLM as comparable to or better than GPT-3.5 and GPT-4, while usually favoring Claude-2; GPT-4 instead preferred InvestLM.The expert and GPT-4 evaluations are presented in Figures 1 and 2.Source: §4 · p. 4, Figure 1 · p. 5, Figure 2 · p. 5
• Expert–expert agreement was 72.5% under a tie-discounted accuracy measure, which assigns half credit when one annotator labels a response a tie.The authors describe this as a reasonable level of agreement despite the subjectivity of response classification.Source: §4 · p. 5

4.1 InvestLM vs. LLaMA response

• In a rate-news example, LLaMA introduced unsupported market figures, while InvestLM gave a reasoned answer that acknowledged uncertainty and investment risks.InvestLM noted that a recommendation depends on the individual’s situation and risk tolerance.Source: §4.1 · p. 5

5 How does InvestLM generalize to financial NLP tasks?

• Financial sentiment, document classification, numerical reasoning, and summarization benchmarks comprise the evaluation; four datasets were developed by the authors.The author-developed datasets are FinSent, ESG, FLS, and QA; the other listed datasets are publicly available.Source: §5 · pp. 5–6

5.1 Generalizability on financial NLP tasks

• InvestLM outperforms LLaMA-65B on 8 of 9 tasks, while GPT-4 leads on 6 and InvestLM on 2.InvestLM was not explicitly tuned on these tasks; its instruction dataset contains around 1,300 examples.Source: §5.1 · p. 6, Table 3 · p. 6

5.2 The benefit of domain instruction tuning

• 138.4% average improvement for LLaMA-7B contrasts with 28.2% for LLaMA-65B after domain instruction tuning.The authors say tuning the smaller model is valuable when computational constraints prevent deploying a 65B model.Source: §5.2 · p. 6, Table 4 · p. 6

5.3 Do generic instructions help?

• InvestLM-7B trained with domain instructions consistently outperforms its variant augmented with 52K Alpaca instructions across all benchmark tasks.The authors interpret this as evidence for carefully curated, high-quality domain instructions over a larger volume of generic instructions.Source: §5.3 · p. 7, Table 5 · p. 7

Financial Domain LLMs

• Earlier financial-domain language models such as FinBERT used BERT architecture before ChatGPT’s release.Source: §6 · p. 7
• BloombergGPT is a 50B foundation model trained on Bloomberg internal corpora but is not publicly accessible (Wu et al., 2023).FinMA instruction-tunes LLaMA using public NLP benchmarks, while FinGPT was fine-tuned on news and tweets sentiment data (Xie et al., 2023; Yang et al., 2023).Source: §6 · p. 7
• InvestLM’s manually curated financial instructions cover a wider range of topics than FinMA’s benchmark-formatted data and FinGPT’s sentiment-focused data (Xie et al., 2023; Yang et al., 2023).The authors describe InvestLM as, to their knowledge, the first open-source financial domain LLM providing insightful investment-related responses.Source: §6 · p. 7

Instruction Tuning

• General instruction-tuning datasets include Natural Instructions, Flan, and OIG; LLaMA-based examples include Alpaca and Vicuna (Mishra et al., 2021; Longpre et al., 2023).Source: §6 · p. 7

• InvestLM instruction-tunes LLaMA-65B with manually crafted instructions covering diverse financial and investment topics.Experts rate its responses as comparable to state-of-the-art commercial LLMs.Source: §7 · p. 8

• InvestLM responses are not definitive investment advice; users should consider their financial situation, risk tolerance, and goals, consult advisors, and use multiple sources.Source: Ethics Statement · p. 8

A Appendix A: Training Format

• The training format takes instructions and input as the basis for predicting an output.Source: A Appendix A: Training Format · p. 9
• The “prompt with input” template pairs an instruction with further context and asks for a response completing the request.Source: A Appendix A: Training Format · p. 9
• The “prompt without input” template asks for a response to an instruction without an input field.Source: A Appendix A: Training Format · p. 9

B Appendix B: Expert Evaluation Interface and GPT-4 Evaluation Prompt

• The GPT-4 prompt was adjusted to ignore its reluctance to make financial recommendations.Source: B Appendix B: Expert Evaluation… · p. 9
• Figure 3 presents the GPT-4 evaluation prompt, as identified by its caption; the cited passage does not state a pairwise-choice protocol or describe the evaluation procedure.Source: Figure 3 · p. 10
• Figure 4 shows the interface used to collect expert preference judgments, with scenario simulations and moderate guidance for experts.Source: B Appendix B: Expert Evaluation… · p. 9, Figure 4 · p. 10