PaperlayerView exact revision on arXiv

Source-linked AI summary

Designing Heterogeneous LLM Agents for Financial Sentiment Analysis

Frank Xing

arXiv:2401.05799v1cs.CLcs.AIcs.MAq-fin.GN

TL;DR

FSA lacks strong guidance for using generative LLMs without fine-tuning, despite the importance of accurate financial sentiment. This paper proposes theory-informed heterogeneous LLM agents specialized around FSA error types and aggregates their discussions. HAD generally improves accuracy across datasets, especially with substantial discussions, while remaining more computationally intensive than naive prompting.

  • Problem

    Generative LLMs are powerful but remain underexplored for FSA, where discriminative task demands and unclear prompt-design guidance limit prescriptive knowledge.

    Source: §2.2 · p. 4

  • Method

    HAD uses five heterogeneous, error-specialized LLM agents grounded in Minsky’s theory of mind and aggregates their analyses for sentiment classification.

    Source: §1 · pp. 2–3

  • Results

    HAD improves FSA accuracies across existing datasets and fixes around 25% – 35% of the performance gap between prompting and fine-tuning, especially with substantial discussions.

    Source: §5 · p. 11

  • Takeaways & Limitations

    The zero-shot, training-free framework provides a theory-informed design option for researchers and financial practitioners building or selecting FSA systems.

    Source: §1 · p. 3, §5 · p. 12

    HAD is slower and more costly than statistical analysis, and possible exposure of evaluation datasets to LLM training data cannot be excluded.

    Source: §5 · p. 12

Abstract

from arXiv · show

Large language models (LLMs) have drastically changed the possible ways to design intelligent systems, shifting the focuses from massive data acquisition and new modeling training to human alignment and strategical elicitation of the full potential of existing pre-trained models. This paradigm shift, however, is not fully realized in financial sentiment analysis (FSA), due to the discriminative nature of this task and a lack of prescriptive knowledge of how to leverage generative models in such a context. This study investigates the effectiveness of the new paradigm, i.e., using LLMs without fine-tuning for FSA. Rooted in Minsky's theory of mind and emotions, a design framework with heterogeneous LLM agents is proposed. The framework instantiates specialized agents using prior domain knowledge of the types of FSA errors and reasons on the aggregated agent discussions. Comprehensive evaluation on FSA datasets show that the framework yields better accuracies, especially when the discussions are substantial. This study contributes to the design foundations and paves new avenues for LLMs-based FSA. Implications on business and management are also discussed.

1 Introduction

FSA is increasingly important for financial decision-making, but generative LLMs remain underexplored because the task is discriminative and lacks prescriptive design guidance. HAD addresses this gap with specialized agents grounded in FSA error types and reports generally improved performance.

Motivation

  • FSA matters to companies, investors, and other stakeholders because sentiment supports return and volatility forecasting, fraud and fake-news detection, and risk prediction.The paper situates accurate FSA within increasingly digitalized financial disclosures and online investor communication.Source: §1 · p. 1

Research gap

  • Generative LLMs remain an early and sparsely studied direction for FSA, whose strongest prior systems mainly fine-tune encoder models such as BERT and FinBERT.The paper explores decoder-based generative LLMs for classification because current FSA research has limited in-depth study of this direction.Source: §1 · p. 2

Contribution

  • HAD uses five specialized LLM agents that simulate different mental resources, discuss a shared message, and aggregate their outputs for FSA.The framework is rooted in Minsky’s theory of mind and emotions and specializes agents through prompts targeting error types.Source: §1 · p. 2

Research questions

  • The study asks whether HAD outperforms naive prompting and fine-tuning, how to induce heterogeneous behavior, and how much each agent contributes.These questions frame effectiveness, prompting, and relative agent importance as separate evaluation targets.Source: §1 · p. 3

Evaluation

  • Evaluation combines performance metrics on five FSA datasets, ablations, and case studies of outputs and intermediary representations.Results indicate generally improved FSA performance, with substantial discussions and attention-guiding templates affecting agent behavior.Source: §1 · p. 3

2 Related Work and Design Process

Prior FSA and prompt-design research shows that general sentiment methods and heuristic LLM collaboration do not fully address finance-specific reasoning or provide strong design theory. The paper therefore grounds heterogeneous agents in Minsky’s theory of mind and formalizes a discussion-based framework.

2.1 Using LLMs for Financial Sentiment Analysis

  • FSA performance can fall relative to general sentiment analysis because financial language relies heavily on domain-specific terminology and linguistic features.The section positions FSA as a business-oriented application requiring finance-sensitive evaluation.Source: §2.1 · p. 3
  • Single-LLM strategies use retrieval, chain-of-thought, or multi-hop reasoning, but FSA requires several capabilities and existing designs remain largely heuristic.These capabilities include reasoning, fact-checking, and syntactic or semantic parsing.Source: §2.1 · p. 4
  • LLM collaboration methods assign auxiliary tasks or roles, yet their designs lack solid theoretical foundations and have limited direct evidence in FSA.The related work contrasts these approaches with HAD’s theory-informed design.Source: §2.1 · p. 4

2.2 Prompt Engineering

  • FSA prompt guidelines remain unclear, with many studies using naive templates rather than domain-specific reasoning knowledge.Examples include generic sentiment-assistant prompts and direct positive/negative/neutral questions.Source: §2.2 · p. 4

2.3 Kernel Theory: Emotions and the Society of Mind

  • Minsky’s society-of-mind theory treats intelligence as interaction among simpler resources and emotions as patterns of resource activation and suppression.This formative account is selected because the design requires a theory explaining how emotions emerge.Source: §2.3 · p. 5

2.4 Meta-requirements, Meta-designs, and Hypotheses

  • HAD simulates resources with prompted LLM agents that all receive the original message, specialize their functions, and contribute analyses that are concatenated for aggregation.Specialization also makes resource activation plausible because agents should respond less meaningfully to out-of-scope contexts.Source: §2.4 · p. 5
  • The framework’s hypotheses predict improved accuracy and F-1 scores, with unequal but positive contributions from agents because error frequencies vary across domains.The design therefore evaluates both overall performance and relative agent importance.Source: §2.4 · p. 6

3 Design Artifact: Heterogeneous Agent Discussion (HAD)

HAD instantiates five agents from finance-specific error categories, gives each a targeted prompt, and combines their opinions through a final sentiment prompt. The design operationalizes heterogeneous attention rather than merely assigning different roles.

Error taxonomy

  • The agent categories derive from documented sentiment-analysis errors including brevity, imbalance, temporal dependency, rhetoric, dependent opinion, unspecified aspects, and external reference.The design uses Xing et al.’s six FSA-relevant categories as its main operational basis.Source: §3 · p. 6

Agent design

  • Five agents are selected from six FSA error categories because they are more finance-relevant and operational; unrecognized words receive no dedicated agent.The omitted category is excluded because LLMs are considered robust to web spelling and unrecognized-word variation.Source: §3 · p. 6

Agent prompts

  • The mood, rhetoric, dependency, aspect, and reference agents target irrealis mood, rhetorical devices, speaker sentiment, the relevant entity, and implicit contextual facts.Their prompts direct attention to distinct error sources including sarcasm, third-party sentiment, ticker or topic mismatch, and time or price references.Source: §3 · p. 6

Aggregation

  • The summative prompt asks for positive, negative, or neutral sentiment while incorporating the test message and additional expert opinions.The wording is adjusted for binary versus ternary testbeds.Source: §3 · p. 6

4 Evaluation

HAD is evaluated through empirical testing, ablation analysis, and case studies across five financial sentiment-analysis datasets. It generally improves performance, with heterogeneous-agent contributions varying substantially and discussions correcting several naive-prompting errors.

4.1 Performance Improvement

  • The evaluation uses five datasets spanning financial news, headlines, and social media, with ternary tasks assessed using macro F-1 scores.FiQA scores are filtered and mapped to positive or negative classes for consistent classification evaluation.Source: §4.1 · p. 8
  • +2.24% to +9.46% accuracy and +0.35% to +13.72% F1-score are reported for HAD on GPT-3.5 across the evaluation datasets.HAD’s effect on BLOOMZ is minimal except on StockSen, where improvements are approximately +10%.Source: Table 3 · p. 8, §4.1 · p. 9
  • HAD is estimated to close 25%–35% of the performance gap between in-context learning and fine-tuning across four datasets.The comparison is presented as an estimate against differences between GPT-3.5 and (Fin)-BERT.Source: §4.1 · p. 9

4.2 Ablation Analysis

  • Removing mood (A1), rhetoric (A2), or aspect (A4) agents generally reduces performance, while reference-agent effects vary and dependency-agent removal can improve results.These patterns indicate complicated non-linear interactions and leave room for further design optimization.Source: §4.2 · p. 9, Table 4 · p. 9

4.3 Case Study

  • Case studies show HAD correcting naive-prompting errors involving company context, mixed sentiment, commodity interpretation, jargon, and uncertain price movement.The framework’s final decisions follow aggregated heterogeneous-agent judgments, including majority neutral predictions for uncertain language.Source: §4.3 · pp. 10–11

5 Discussion, Conclusion, and Future Work

HAD is a theory-informed, zero-shot framework that improves financial sentiment analysis by coordinating heterogeneous LLM agents specialized around known error types. Its benefits span existing datasets but involve higher computational costs and unresolved concerns about dataset confidentiality and remaining errors.

  • HAD improves FSA accuracies across existing datasets, fixing around 25%–35% of the performance gap between prompting and fine-tuning.The framework is especially effective when agents produce substantial discussions.Source: §5 · p. 11
  • HAD is a zero-shot, training-free collaboration framework that applies heterogeneous error-type-specialized agents to financial sentiment analysis.It is less complex than many other LLM collaboration and fine-tuning-based approaches.Source: §5 · pp. 11–12
  • Mood, rhetoric, and aspect agents are more important than the reference agent, while removing the dependency agent can further optimize performance.The evaluation supports H1 but rejects H2.Source: §5 · p. 11
  • The framework could help financial advisors, traders, fund managers, and investors build in-house sentiment-analysis tools or explore FSA system designs.These practical implications follow from the framework’s training-free design.Source: §5 · p. 12
  • Scaling HAD is constrained by slower, costlier agent discussions, and prior exposure of evaluation datasets to LLM training cannot be excluded.The study also calls for research on remaining LLM errors and human- or expert-level performance.Source: §5 · p. 12
Loading 2401.05799v1…