Source-linked AI summary
Transforming Sentiment Analysis in the Financial Domain with ChatGPT
Georgios Fatouros, John Soldatos, Kalliopi Kouroumali, Georgios Makridis, Dimosthenis Kyriazis
TL;DR
Financial sentiment analysis must handle specialized, multi-instrument, and context-dependent language, yet ChatGPT’s API-based use in forex sentiment analysis remained underexplored. This study evaluates ChatGPT 3.5 with zero-shot prompts on curated forex headlines, finding performance above FinBERT and emphasizing prompt selection. The authors also note that sentiment does not fully explain market-price variation and that response-time variability requires further testing.
Problem
Financial sentiment analysis faces specialized terminology, multiple instruments, and context-dependent sentiment, while ChatGPT’s forex application remained underexplored.
Method
The study evaluates ChatGPT 3.5 with zero-shot prompts on a curated, annotated dataset of forex-related news headlines, made publicly available.
Results
ChatGPT’s performance exceeds FinBERT when prompts accurately frame short-term sentiment classification, with strategic prompt selection highlighted as important.
Takeaways & Limitations
Prompt selection is important for optimizing ChatGPT’s financial sentiment-analysis performance in zero-shot settings.
Takeaways & Limitations
Sentiment does not fully align with market movements, indicating that it explains only part of market-price variation.
Abstract
from arXiv · showhide
Financial sentiment analysis plays a crucial role in decoding market trends and guiding strategic trading decisions. Despite the deployment of advanced deep learning techniques and language models to refine sentiment analysis in finance, this study breaks new ground by investigating the potential of large language models, particularly ChatGPT 3.5, in financial sentiment analysis, with a strong emphasis on the foreign exchange market (forex). Employing a zero-shot prompting approach, we examine multiple ChatGPT prompts on a meticulously curated dataset of forex-related news headlines, measuring performance using metrics such as precision, recall, f1-score, and Mean Absolute Error (MAE) of the sentiment class. Additionally, we probe the correlation between predicted sentiment and market returns as an additional evaluation approach. ChatGPT, compared to FinBERT, a well-established sentiment analysis model for financial texts, exhibited approximately 35\% enhanced performance in sentiment classification and a 36\% higher correlation with market returns. By underlining the significance of prompt engineering, particularly in zero-shot contexts, this study spotlights ChatGPT's potential to substantially boost sentiment analysis in financial applications. By sharing the utilized dataset, our intention is to stimulate further research and advancements in the field of financial services.
1. Introduction
Financial sentiment analysis faces domain-specific, contextual challenges that can make conventional models misleading. This study addresses these challenges by evaluating ChatGPT 3.5 with zero-shot prompts on curated forex headlines and sharing the annotated dataset for further research.
- Motivation: Financial headlines combine specialized terminology, nuanced sentiment, and multiple instruments, making sentiment classification potentially misleading without identifying the sentiment’s subject.The CAD and USD/CAD example illustrates how sentiment can differ across related financial instruments.
- Motivation: Conventional sentiment models may not adjust outputs to different use-case contexts, limiting their applicability across financial settings.Investor and consumer discussions of the same regulatory policy may express different relevant sentiments.
- Study approach: ChatGPT 3.5 is evaluated through zero-shot prompting for financial sentiment analysis focused on the foreign exchange market.The approach avoids domain-specific fine-tuning while assessing ChatGPT’s ability to interpret financial text.
- Study approach: The study uses a curated, annotated dataset of forex-related news headlines and makes it publicly available for future research.The dataset supports evaluation of multiple ChatGPT prompts and further work in financial services.
- Study approach: Multiple prompts are evaluated for sentiment class prediction and sentiment score estimation on single and multiple headlines using classification, error, and market-return measures.The evaluation includes precision, recall, f1-score, Mean Absolute Error, and correlation with market returns.
- Study approach: Prompt engineering, especially in zero-shot settings, is examined as a way to optimize financial sentiment-analysis performance.The study benchmarks prompts to assess their effect on efficacy and adaptability.
2. Related Work
Financial sentiment analysis evolved from lexicons and traditional machine learning to contextual deep-learning language models, while finance-specific challenges remain. ChatGPT’s direct API application to forex sentiment analysis was identified as an underexplored research area addressed by this study.
- Sentiment analysis in finance: Financial sentiment analysis extracts subjective information from text to gauge market sentiment that may affect financial-instrument prices.The concept is applied to sources including news articles and other textual data.
- Sentiment analysis in finance: Early financial sentiment analysis used manually labeled lexicons, but this approach could overlook the context in which words are used.A word such as “crisis” can carry different sentiment depending on its surrounding statement.
- Sentiment analysis in finance: Machine-learning models improved contextual classification, while traditional methods remained limited in capturing long-term textual dependencies.The related work describes this progression from labeled financial articles to more advanced language models.
- Open challenges: Financial sentiment analysis still requires domain-specific understanding, disentangling sentiments across instruments, and adaptation to use-case context.These challenges motivate investigating models with stronger contextual understanding and adaptability.
- Related language models: BloombergGPT demonstrated financial NLP capabilities but lacked an open API and was primarily used internally by Bloomberg as of May 2023.This limited its direct availability for broader application integration at that time.
- Related language models: BLOOM was open source but required specialized knowledge and substantial computational resources, without a conversation-tuned version.These requirements distinguished it from conversational models such as ChatGPT.
- Research gap: Direct API-based evaluation of ChatGPT for financial sentiment analysis, particularly in forex, was described as largely absent from prior research.The study positions its API interaction as relevant to deployment in third-party applications and services.
3. Methodology
The methodology builds and annotates a forex-news dataset, then evaluates ChatGPT through zero-shot prompts against FinBERT using sentiment and market-oriented measures.
- Dataset Creation and Annotation: The study collects forex headlines for AUDUSD, EURCHF, EURUSD, GBPUSD, and USDJPY from Forex Live and FXstreet over 86 days.The dataset covers January to May 2023.
- Dataset Creation and Annotation: The dataset contains 2,291 unique headlines with associated forex pairs, timestamps, sources, authors, URLs, and article text.Collection used web scraping through a custom virtual-machine service.
- Dataset Creation and Annotation: Each headline is manually labeled positive, negative, or neutral according to its potential short-term impact on the corresponding forex pair.The categories correspond to bullish, bearish, and hold sentiments, respectively.
- Baseline Model: ChatGPT is compared with FinBERT, integrated through Hugging Face Transformers as a conventional financial-sentiment baseline.FinBERT tokenizes headlines and outputs sentiment probabilities.
- ChatGPT Evaluation: ChatGPT is evaluated with zero-shot prompts spanning analyst, sentiment-model, trader, forex-pair, and aggregate-news perspectives.The experiments use GPT-3.5-turbo, with prompt variations for sentiment classification and numeric outputs.
- Evaluation: Evaluation includes sentiment metrics, S-MAE, and Directional Accuracy for agreement between predicted sentiment direction and actual market-return direction.S-MAE uses integer sentiment classes, while Directional Accuracy is the percentage of correctly matched movement directions.
4. Results
GPT models generally outperform FinBERT in forex sentiment classification, while prompt design and forex-pair context materially affect performance. Numerical sentiment models also show useful market-direction results, but outcomes vary across currency pairs and processing conditions.
- Sentiment Classification Performance: GPT models consistently outperform FinBERT across all sentiment-classification metrics.FinBERT records 0.561 accuracy, 0.560 precision, 0.562 recall, 0.556 F1-Score, and 0.540 S-MAE.
- Sentiment Classification Performance: GPT-P1 trails GPT-P2 by approximately 8%, indicating that the sentiment-oriented P2 approach is more precise than P1’s emotion-based framing.Both prompts incorporate the related forex pair in their analysis.
- Sentiment Classification Performance: GPT-P2 leads sentiment classification with accuracy, recall, and F1-Score around 0.790, while GPT-P4 performs similarly under a different prompt framing.The results align with the dataset’s annotation approach and support the effectiveness of role-playing prompts.
- Forex-Pair Effects: Performance varies by forex pair: GPT-P2 dominates several metrics for AUDUSD and USDJPY, whereas GPT-P4 dominates EURUSD and GBPUSD.For AUDUSD, GPT-P4 leads precision while GPT-P2 leads accuracy, recall, F1-score, and S-MAE.
- Robustness Without Explicit Pair Context: When the forex pair is omitted, GPT-P2 still records 0.711 accuracy, 0.728 precision, 0.711 recall, 0.714 F1-score, and 0.326 S-MAE, exceeding FinBERT.FinBERT’s corresponding values are 0.543, 0.543, 0.543, 0.538, and 0.539.
- Directional Accuracy: GPT-P1N achieves the highest directional accuracy at 67.2%, while numerical models generally exceed categorical counterparts.GPT-P6N leads for EURUSD and GBPUSD, whereas FinBERT-N leads for USDJPY, showing ticker-specific strengths.
5. Discussion
The study finds that ChatGPT generally outperforms FinBERT in financial sentiment analysis, while prompt design and aggregation of daily news affect performance and market-return alignment. The authors emphasize practical potential alongside limits involving prompt variability, market complexity, and operational response time.
- ChatGPT consistently outperformed FinBERT across the evaluated prompts, with GPT-P4, P6, and P6N showing particularly strong performance.
- GPT-P6N achieved higher correlation with market returns than the actual sentiment, suggesting stronger capture of aggregate market sentiment when processing daily news together.
- N-version models producing sentiment scores from -1 to 1 aligned more closely with market movements, whereas GPT-P5 showed lower correlation with market returns and true sentiment.
- Robust correlations across GPT-P1/P1N through GPT-P6/P6N support the consistency of GPT outputs and their promise for real-world applications.
- Strategic prompt selection may provide actionable insights for predicting market trends, but the best prompt depends on the use case, financial instrument, and tolerated prediction error.
- Alignment between sentiment scores and market movements does not establish predictive power over future prices because markets reflect macroeconomic, geopolitical, and technical factors.
- Sentiment explains only part of price variation, and further research should compare emerging models and assess response-time variation under different operating conditions.
- The findings indicate significant potential for ChatGPT in financial sentiment analysis while motivating further work toward more sophisticated and accurate models.
6. Conclusion
The conclusion presents a zero-shot evaluation of ChatGPT 3.5 for forex sentiment analysis using a curated, publicly released news-headline dataset. It reports that task-aligned prompting can outperform FinBERT, while emphasizing prompt selection, further data integration, and operational robustness as priorities for future work.
- ChatGPT 3.5 was evaluated through zero-shot prompting on a curated and annotated dataset of forex-related news headlines made publicly available for further research.
- Prompts that accurately frame short-term sentiment classification showed potential for understanding market behavior and exceeded the performance of FinBERT.
- Strategic prompt selection enhanced sentiment-analysis performance and efficiency, while ChatGPT remained robust and consistent across prompts.
- Processing all daily news at once opens future research into integrating additional relevant financial data within prompts.
- The findings substantiate the potential relevance of large language models in finance, including the possibility that newer models such as GPT-4 may improve performance.
- Real-world application remains challenged by varying time performance under different load conditions, requiring more accurate and consistent performance measures.
- The study contributes a comprehensive evaluation and released dataset intended to support further refinement of LLMs for financial services.
CRediT authorship contribution statement
The authors contributed across conceptualization, data, analysis, methodology, software, validation, supervision, writing, funding, administration, and resources.
- Georgios Fatouros led conceptualization, data curation, formal analysis, investigation, methodology, software, validation, visualization, and original drafting.
- John Soldatos contributed conceptualization, funding acquisition, supervision, and manuscript review and editing.
- Kalliopi Kouroumali contributed data curation, while Georgios Makridis contributed manuscript review.
- Dimosthenis Kyriazis contributed funding acquisition, project administration, and resources.