Stock trend prediction using news sentiment analysis

1. INTRODUCTION

• Volatility and the volume of current and past market information make forecasting a stock’s next trend difficult for an individual.The paper presents stock forecasting as a way to help investors and analysts plan buying or selling strategies.Source: §1 · p. 1
• The project treats company news as fundamental information and classifies articles as positive or negative to study their relationship with stock trends.The authors assume positive news may accompany a price rise and negative news may accompany a decline.Source: §1 · p. 2
• Supervised machine-learning and text-mining techniques classify news polarity, including for articles not used to build the classifier.The project implements three classification algorithms to compare and improve classification accuracy.Source: §1 · p. 2

2. LITERATURE SURVEY

• The survey links accurate stock predictions with returns and describes stock-trend forecasting as an active research area.It notes efforts to develop models for predicting trends in specific stocks or the overall market.Source: §2 · p. 2
• The survey says many existing techniques use technical indicators and cites research reporting a strong relationship between company news and stock-price fluctuations.Source: §2 · p. 2
• Prior work proposed NewsSentiment, a positive- and negative-word-count measure whose time variation was reported to correlate strongly with stock-price movement.The cited work also developed news collection and sentiment evaluation engines.Source: §2 · p. 2
• Yu et al quantified news sentiment as a time series and compared it with energy demand and price fluctuations.Source: §2 · p. 3
• J. Bean used keyword tagging to score airline-related Twitter feeds for sentiment and customer satisfaction.Source: §2 · p. 3
• A financial forecasting study found that using five news-relevance categories together improved prediction performance over using fewer categories.The categories ranged from the target stock’s sub-industry to its sector; the highest accuracy and return per trade were reported with all five categories and polynomial and Gaussian kernels.Source: §2 · p. 3

3.1. System Design

• The system has three phases: produce news articles with polarity scores, classify TF-IDF representations, then compare news and stock-price data.The design evaluates classifiers on new articles and plots news and price data using R.Source: §3.1 · p. 4

3.1.1. News Collection

• The dataset contains Apple Inc. news and daily AAPL prices from February 1, 2013, to April 2, 2016, using Adjusted Close as the daily price.News came from major aggregators, including Google News and Yahoo Finance.Source: §3.1.1 · p. 4

3.1.2. Pre Processing

• Preprocessing tokenizes and cleans news text, removes noisy terms and stop words, filters terms appearing in fewer than three documents, and stems words.The stop-word list combines finance-specific and general English terms.Source: §3.1.2 · p. 5

3.1.3. Sentiment Detection Algorithm

• A dictionary-based sentiment algorithm scores each article by matching its headline and body against positive and negative word lists.The lists include general polarity words and finance-specific words, with the finance polarity drawn from McDonald’s research.Source: §3.1.3 · p. 5

2. Prepare the dictionary which contains words with its polarity (positive or negative)

• The polarity dictionary counts positive and negative matches, subtracts negative matches from positive matches, and labels scores of zero or more positive.A zero score is assigned positive because the implementation uses a two-class problem.Source: §2 · p. 6

3.1.4. Document Representation

• The document representation uses TF-IDF, assigning higher weight to terms frequent in a document but present in fewer collection documents.Common terms receive very low values.Source: §3.1.4 · p. 6

3.1.5. Classifier Learning

• The study compares SVM, Random Forest, and Naive Bayes using accuracy, precision, recall, and other evaluation measures, with implementation in Weka.Source: §3.1.5 · p. 6

3.1.6. System Evaluation

• Evaluation uses train and test data plus an unknown dataset to check classifier accuracy on new data, measuring accuracy, precision, recall, and ROC curve area.Source: §3.1.6 · p. 6

3.1.7. Testing with new Data

• The unknown test set consists of news articles from January through April 2016.Source: §3.1.7 · p. 7
• On unknown news data, SVM performed well, and Random Forest worked well compared with Naive Bayes.Source: §3.1.7 · p. 7

3.1.8. Plotting the values

• After classifying unknown data, the study plots news scores against the historical stock-price chart.Source: §3.1.8 · p. 7

• The listed cross-validation options are 5-fold, 10-fold, and 15-fold cross-validation.Source: §4 · p. 7
• Figure 2 compares three classifiers across different test options.Source: Figure 2 · p. 7
• Figure 3 presents model testing results with new data.Source: Figure 3 · p. 8
• Figure 4 plots news sentiment score against actual stock price for the test dataset.Source: Figure 4 · p. 8

• The study assumes news articles and stock prices are related and that news may affect stock trends.Source: §5 · p. 8
• The conclusion states that stock trends can be predicted using news articles and previous price history, with article polarity used to predict the trend.Source: §5 · pp. 8–9
• The approach uses positive and negative sentiment dictionaries built from general and finance-specific words, along with finance-specific stop words for text preprocessing.Source: §5 · p. 9
• The authors implemented three classification models and tested them under different scenarios before comparing their results.Source: §5 · p. 9
• Random Forest achieved 88% to 92% accuracy across test cases, compared with around 86% for SVM and around 83% for Naive Bayes.Source: §5 · p. 9

• The authors plan to add data from more companies and evaluate prediction accuracy.Source: FUTURE WORK · p. 9
• For companies with limited financial-news availability, they propose using Twitter data for similar analysis.Source: FUTURE WORK · p. 9

• One author is a Master of Engineering student at K. J. Somaiya College of Engineering who completed a bachelor's degree at Pune University in 2013.Source: Authors · p. 10
• Prof. Bharathi H. N. is Head of the Computer Engineering Department at K. J. Somaiya College of Engineering.Source: Authors · p. 11
• Prof. Jyothi M. Rao is an associate professor and associate head of the Computer Engineering Department at K. J. Somaiya College of Engineering.Source: Authors · p. 11