Predicting Stock Market Indicators Through Twitter “I hope it is not as bad as I fear”

We have been working on trying to predict market indicators for quite some time by analyzing Web Buzz, predicting who will win an Oscar, or how well movies do at the box office. Among other things we have correlated posts about a stock on Yahoo Finance and Motley’s Fool with the actual stock price, predicting the closing price of the stock on the next day based on what people say today on Yahoo Finance, on the Web and Blogs about a stock title.

The rising popularity of twitter gives us a new great way of capturing the collective mind up to the last minute. In our current project we analyze the positive and negative mood of the masses on twitter, comparing it with broad stock market indices such as Dow Jones, S&P 500, and NASDAQ. We collected the twitter feeds from one whitelisted IP for six months from March 30, 2009 to Sept 4, 2009, ranging from 5680 to 42820 tweets per day. According to twitter this corresponds to a randomized subsample of about one hundredth of the full volume of all tweets, as the total volume in 2009 was about 2,5 million tweets per day. We tried to measure collective hope and fear on each day by applying the simple metric of counting all tweets containing the words “hope” – there were 54 to 467 tweets per day, and “fear” or “worry” – there were 9 to 100 tweets per day. This tells us that people prefer optimistic words (hope) to pessimist words (fear or worry).


As external benchmark of investor fear we used the Chicago Board Options Exchange Volatility Index VIX, which is strongly negatively correlated with Dow, S&P 500, and NASDAQ, which is not surprising, as the spread of stock options on a given day is used to calculate VIX. Initially we expected the number of tweets with hope to negatively correlate with VIX, and the number of tweets with fear or worry to correlate positively with VIX. Surprisingly, we found positive weak but insignificant correlation for both “hope” (0.135) and “fear” or “worry” (0.172) with VIX, and negative significant correlation with both “fear” and “worry” and “hope” with Dow NASDAQ and S&P500 (This means that people start using more emotional words such as hope, fear, worry in times of economic uncertainty. We therefore created a simple twitter-volatility index combining mentions of hope, fear and worry, normalizing it with the total amount of tweets per day as a baseline. This index displays strong significant negative correlations to Dow, NASDAQ and S&P500, and strong significant positive correlation to VIX (see table below).


The picture below visualized the negative correlation between Dow (blue) and “hope, fear, and worry” (green) in the period March 30, 2009 to Sept 4, 2009.


To put this in simple words, when the emotions on twitter fly high, that is when people express a lot of hope, fear, and worry, the Dow goes down the next day. When people have less hope, fear, and worry, the Dow goes up. It therefore seems that just checking on twitter for emotional outbursts of any kind gives a predictor of how the stock market will be doing the next day.

Just to be clear, what we have presented here are very early preliminary results, and much more work is needed to scientifically verify it.