Hello. I wish to try developing some trading algorithms in Matlab (because I am familiar with it and like it). I presume I need a source to download data. I don't plan to make real trades at this point. I would need to access download for complete NYSE, NASDAQ, currencies, commodities, and everything else. I am a beginner when it comes to trading and investing. Are there other things I need (I know if I start actually trading I will need a broker, etc.) for just developing algorithms and testing them on data from the past up to the present? Thank you very much!
www.money.net has an api for realtime data into matlab. Otherwise try the free data api at https://www.quandl.com/tools/api
You effectively said you want all market data. I think you substantially underestimate how much data that is and how many different assets are out there. You also didn't provide a time frame just a generic "past up to the present". Can you start with what sort of resolution you're looking for and what time frame you're looking for? What I mean is, do you want daily/hourly/minute/second/tick? Do you want OHLC data, all trades, NBBO quotes, quotes from specific exchanges, all quotes across all exchanges? You mentioned NYSE and NASDAQ, do you only care about US these exchanges or do you care about other US exchanges? Do you care about non-US markets? Do you want for the last month, last year, last decade? What's your budget (ie what do you THINK all this data is going to cost)? This is partially for my amusement, but it's also to make you realize that most data isn't free and if you put a number on it we can offer you some suggestions that are at least reasonable given your budget. Based on what I think your answer is going to be, you better order a few backblaze 60 drive storage pods and get those up and running. I think you need to do more research on the different types of assets traded and define the time period you care about (ie last 10 years). You also need to determine a resolution (ie anything from daily OHLC to every trade and quote on every exchange). Low resolution data (like daily OHLC) can often be found for free by scraping different sites or low cost. When you get in to higher resolution (ie minute, all trades, all quotes) you get in to high costs and massive amounts of data. We're not talking on the order of a few MB compressed per day here, we're talking on the order of 10s of GB per day of data and that's just for US stock, stock options, futures, and futures options. You also mention currencies, well that one is going to be tough. There's no centralized currency exchange to acquire data from because it's an OTC product.
Lee, thanks a lot for the reply. I got an inkling of how much data it is when I looked at the sites just21 posted. But 10gb/day...wow. I'm hoping for free, which Quandl has. I have discovered CSI which has 5 years of stock market daily data including current daily data for about 30 a month. Or 20 years of historical for 200 each (NYSE, NASDAQ, AMEX). Yes that's ~10,000 stocks. I don't expect this data to be fine enough resolution for day trading strategies. Do you know of a site where I can check out prices for that? Thanks a lot, much appreciated.
If you're looking for daily data, that's not so bad in terms of cost and size. You can actually scrape a lot of it for free. For example, yahoo finance has free daily historical data and there are some open source scrapers for their data (I've not used any of them, so can't recommend any). For high resolution historical data, there are data vendors, but they're all expensive. You don't necessarily need every quote/trade for intra-day. You could look at 1 minute OHLC bars for example. One of the cheaper sources I've found that claims to be rather complete (but I've not personally used them and have no idea how good their data actually is) is kibot -- http://www.kibot.com/buy.aspx -- their "ultimate" package is $7900 and is 5TB uncompressed. That seems a bit low to me, so I'm assuming it does not include options. For a more likely to be good data, you can go straight to the exchange, but it's going to be even more money. For example, for futures one of the major organizations is CME. You can buy data direct from CME for the exchanges they operate: https://datamine.cmegroup.com/#t=p&p=cme.eod You could optionally switch it to the individual futures you're interested in by selecting "Individual products" on the left. You add whatever you want to cart (including date ranges). Make sure you're sitting down before you look at the cart. Other exchanges will similarly have a page to buy historical data from them. When you get to high resolution data, you will probably want NBBO data (National Best Bid Offer, which is the best price across all exchanges) rather than data from just a specific exchange. Be aware, there are many more exchanges in the US than just NYSE/NASDAQ/AMEX. A stock may have its primary listing on NASDAQ (like say TSLA), but for example, my broker lists 14 different exchanges/ecns on which I can trade TSLA. My recommendation would be to just stick with free daily data until you play around with it and become more familiar with the different assets and exchanges out there. You'll probably also get a better idea of what you actually want by doing this and save yourself a lot of time and money.
Thanks again Eric. Yes, I'll have to stick with the free stuff right now, or I will have lost considerable money on this project before I get started. I think you are supposed to lose money after you get started trading . Thanks very much for the great info, very educational for this novice. Maybe some day I will be successful enough that I will need those resources, and able to pay for them . Jim
Thanks dealmaker. I gave them a call, and their historical data is only $10,000. But when I talked to the guy, he told me about a place to get data that is more geared for people like me, called Metastock.com, and I'll give them a call. Or were you interested in staking my investing efforts using the global data? I absolutely guarantee you 10,000% annual returns. Jim
Jim, under trader tax status those kind of costs are write offs... Only 720 trades, 31 days or less holding period would do it.