Big data by data mining and data analytics is hot in business. But it is still said hardly effective in processing financial data such as stock and option prices. Is this true or not? At least, as far as I know, there is no any publicly available good big data algorithms in stock/option trading. However, there are some dark forces in wall street are still building huge powerful super-computers to crunch financial data, bet on finding price movement patterns. Not know how much this contributes to their success or their success is just based on their purely price manipulation.
Stock data is not Big Data unless you use fundamental information too. Dark forces in Wall Street? Impossible. Data mining? This is a good read that provides an idea of what some people do. Here is another good article by Jonathan Kinlay. Note that are many pitfalls in these approaches.
" High-tech traders use social media to gain edge http://www.smh.com.au/business/mark...ocial-media-to-gain-edge-20140827-1092i0.html Greg Daniel, the chief executive of KPMG SR7, a social media intelligence and research firm, says hedge funds around the globe have been ahead of the market in hiring analytical experts who give them a slight edge. ... Hedge funds and high-frequency traders gain a time advantage through their sophisticated monitoring of social media networks and advanced algorithms that can sift through information, predict trends and deliver an edge. "
This is why a The algo bots are using NLP to intreprets tweets and news headlines. whenever the journalist had a typing snafu, you can catch the hilarious results in the financial market. So if somebody hacked twitter and changed bloomber tweet too US DEFAULT there will be some fireworks.
I may scan, filter, run complex algorithms 10-year of stock data of all stocks traded on the market on my personal laptop bought in regular computer store. Computers are not a problem. The problem is to have access to the data and have ideas about what to test/scan what algorithm to run. In many cases the online data/charts providers companies have data but it is not their business to do analysis instead of traders. At the same time traders ho has ideas have limited access to the big amount of historical data as it could be quite costly. And yes, what you called "dark forces", I see why a successful fund managers who has access to the historical stock market data cannot hire developers and analysts to test and run his ideas (algorithms).