Source-linked AI summary
Sentiment Analysis of Twitter Data for Predicting Stock Market Movements
Venkata Sasank Pagolu, Kamal Nayan Reddy Challa, Ganapati Panda, Babita Majhi
TL;DR
The paper asks whether public sentiment in company-related tweets is correlated with stock-price rises and falls, addressing a prediction problem complicated by fluctuating markets and limited predictability from historical prices. It applies sentiment classification with Word2vec and N-gram representations, then uses supervised learning to model stock movements. The reported classifiers achieve accuracies up to 71.82%, and the authors conclude that stock movements strongly correlate with Twitter sentiment.
Problem
The paper examines whether company-related Twitter sentiment correlates with stock-price rises and falls, against a background of fluctuating prices and limited predictability from historical prices.
Method
The study classifies tweets using Word2vec and N-gram representations with supervised machine learning, then uses aggregate three-day sentiment values to classify the next day’s stock-price movement.
Results
71.82% accuracy was achieved with LibSVM using 90% of the data, while Logistic regression achieved 69.01% accuracy and the sentiment classifiers achieved 70.2% with Word2vec and 70.5% with N-grams.
Takeaways & Limitations
The authors conclude that a strong correlation exists between company stock-price rises and falls and public opinions expressed about that company on Twitter.
Takeaways & Limitations
The study uses only Twitter data, which may be biased because not all stock traders share their opinions there, and trains its sentiment analyzer on 3,216 tweets.
Abstract
from arXiv · showhide
Predicting stock market movements is a well-known problem of interest. Now-a-days social media is perfectly representing the public sentiment and opinion about current events. Especially, twitter has attracted a lot of attention from researchers for studying the public sentiments. Stock market prediction on the basis of public sentiments expressed on twitter has been an intriguing field of research. Previous studies have concluded that the aggregate public mood collected from twitter may well be correlated with Dow Jones Industrial Average Index (DJIA). The thesis of this work is to observe how well the changes in stock prices of a company, the rises and falls, are correlated with the public opinions being expressed in tweets about that company. Understanding author's opinion from a piece of text is the objective of sentiment analysis. The present paper have employed two different textual representations, Word2vec and N-gram, for analyzing the public sentiments in tweets. In this paper, we have applied sentiment analysis and supervised machine learning principles to the tweets extracted from twitter and analyze the correlation between stock market movements of a company and sentiments in tweets. In an elaborate way, positive news and tweets in social media about a company would definitely encourage people to invest in the stocks of that company and as a result the stock price of that company would increase. At the end of the paper, it is shown that a strong correlation exists between the rise and falls in stock prices with the public sentiments in tweets.
I. INTRODUCTION
The introduction frames stock prediction as difficult because prices respond to unpredictable news and events, while Twitter provides abundant public opinion that can support sentiment-based analysis. The paper studies Twitter sentiment classification and its relationship to stock movements.
- Historical-price approaches are challenged because stock prices fluctuate and follow a random-walk pattern linked to unpredictable news and current events.The cited passage states that prices cannot be predicted with more than 50% accuracy.
- Twitter offers abundant, real-time public opinion in concise messages, making it a valuable corpus for research and prediction.The passage reports more than 140 million tweets posted daily and describes tweets as concise expressions of public opinion.
- The paper contributes sentiment analysis of Twitter data, classifying opinions in text as positive, negative, or neutral.
- Prior studies used Twitter sentiment and activity to predict movie revenues, disease outbreaks, stock prices, trading volumes, and market indices.
- The paper is organized around data collection and preprocessing, sentiment analysis, stock-sentiment correlation, results, conclusions, and future work.
II. RELATED WORK
Related work established that collective Twitter mood, sentiment, and activity can correlate with market indices, stock prices, and trading volumes. This paper adopts a novel rise-or-fall prediction approach based on sentiment extracted from company-related tweets.
- Earlier studies reported correlations between collective Twitter mood and the Dow Jones Industrial Average, including both strong positive and high negative relationships.
- Researchers also used Twitter feeds to derive investment strategies and found that stock-price predictability varies by industry.
- Twitter activity has been linked to changes in stock prices, trading volumes, and search-query volumes for NASDAQ-100 stocks.
- This paper predicts stock-price rises and falls from Twitter sentiment and develops a sentiment analyzer as its core contribution.
A. Data Collection
The study collects Microsoft-related tweets and stock prices over a common period, then prepares both datasets for sentiment and correlation analysis. Missing stock observations are approximated before modeling.
- 250,000 Microsoft tweets were collected through Twitter’s API from August 31, 2015 to August 25, 2016 using company, product, and service keywords.
- Microsoft opening and closing prices for the same period were obtained from Yahoo! Finance.
- Missing stock-price observations caused by weekends and public holidays were filled by averaging neighboring available values.
- Tweet preprocessing used tokenization, stopword removal, and regular-expression matching to remove special characters and irrelevant content.
1) Tokenization:
Tokenization splits each tweet into individual words, after which stopwords that do not express emotion are removed to produce a cleaner word list.
- Tokenization:: Tokenization divides tweets into individual words using spaces and irrelevant symbols such as emoticons are removed.
- Tokenization:: Stopword removal excludes common words such as “a,” “is,” “the,” and “with” because they do not express emotion.
3) Regex Matching for special character Removal:
Tweets are normalized before sentiment classification by replacing URLs, hashtags, user mentions, and elongated words with standardized forms. The paper also uses a field-specific sentiment analyzer because existing analyzers rely on different corpora.
- URLs are replaced with the token URL during preprocessing.
- Hashtags, user mentions, and prolonged emotional spellings are normalized before classification.Examples include replacing #Microsoft with Microsoft, @Billgates with USER, and coooooooool! with cool!.
- A custom sentiment analyzer classifies tweets as positive, negative, or neutral because movie and stock corpora differ.
A. Feature Extraction
The paper extracts tweet features using n-gram and Word2vec textual representations. This section focuses on n-grams, which build corpus-specific dictionaries from contiguous word sequences.
- The paper uses n-grams and Word2vec for textual feature representation.
- N-gram representation extracts every word sequence of length n from the tweet corpus.
- For a 3-gram representation, “Microsoft is launching a new product” yields four contiguous three-word features.
2) Word2vec Representation:
Word2vec maps words into 300-dimensional vectors, while the resulting sentiment features are classified and then used in stock-movement analysis. The stock task uses preceding three-day sentiment totals to predict the next day’s labeled movement.
- 2) Word2vec Representation:: Word2vec maps each word to a unique 300-dimensional vector, and word vectors can be summed for a text collection.
- 2) Word2vec Representation:: Random forest classifiers receive features from human-annotated tweets; Word2vec is selected for meaning sustainability and promising large-dataset performance.
- V. CORRELATION ANALYSIS OF PRICE AND SENTIMENT: Stock movement is labeled 0 when the current price is lower than the previous day’s price, otherwise 1.
- V. CORRELATION ANALYSIS OF PRICE AND SENTIMENT: The classifier uses total positive, negative, and neutral tweet emotions over three days to predict the next day’s stock label.
- V. CORRELATION ANALYSIS OF PRICE AND SENTIMENT: 355 instances with three attributes are split into 80% training data and the remaining data for testing.
VI. RESULTS AND DISCUSSION
The results section reports classifier accuracy rates. Calculations are performed using Weka, a tool running on a Java virtual machine.
- The section provides an overview of trained-classifier accuracy rates.
- All calculations are performed in Weka.
- Weka runs on a Java virtual machine.
A. Sentiment Analyzer Results
The paper reports sentiment-classification accuracy near 70% and finds a strong correlation between Twitter opinions about a company and its stock-price rises and falls.
- Sentiment Analyzer Results: 70.5% accuracy was achieved by the N-gram classifier, compared with 70.2% for the Word2vec classifier.Both classifiers used Random Forest with the same dataset and training split; Word2vec was selected for classifying nonhuman-annotated tweets.
- Stock Price and Sentiment Correlation Results: 71.82% accuracy was obtained with LibSVM using aggregate three-day sentiment values to predict stock-price increases or decreases.Logistic regression achieved 69.01% accuracy, while the reported accuracy varied with the training set.
- Conclusion: The study reports a strong correlation between stock-price rises and falls and public opinions about the company expressed in tweets.The conclusion links the observed stock-price movements with Twitter opinions or emotions about the company.
- Conclusion: The sentiment analyzer classifies tweets into positive, negative, and neutral categories.Its stated purpose is to judge the sentiment type present in a tweet.
VIII. FUTURE WORK
Future work will broaden the opinion data and enlarge the sentiment-analyzer training set to address current coverage and data-size constraints.
- Future Work: The study uses only Twitter data, which may bias sentiment analysis because not all stock traders share opinions on Twitter.The authors propose incorporating Stocktwits and news data for more exhaustive public-opinion collection.
- Future Work: The sentiment analyzer was trained with 3,216 tweets, and future work aims to human-annotate more than 10,000 tweets.The paper states that increasing the training-dataset size tends to improve model performance.