I've written a custom backtesting program. I have been running it with a mysql database I've filled with data I bought from Pi Trading. That is all fine, but I would like to double check my results with another set of data - just to be careful. I am not entirely confident of this data. Anybody know of a good source for historical 1 minute data for stocks? Tick data would be OK but that gets huge. I would like to have data back at least 5 years. I am willing to pay for it. I have searched the net but have not found the solution. BTW, here's a general "thanks" for all the stuff I've learned from this forum over the years. Creating and running my backtesting and ATS setups is hugely satisifying - monetarily, intellectually and emotionally. Anbody who thinks the long hours, risk tolerance and skills needed for this are easy is a fool. -Kever
YOu can use tickdatamarket.com. They have good discounts when you order a lot. Watch out for pi trading because I saw a lot of errors some years ago. Esspecially in the pre 2005 period. This is the case in most providers. The data is not always clean....
if you want to be dead sure that your data is correct and are willing to pay up for this, either get it direct from the exchange or from a top end data provider such as CQG.
OK, I said I was willing to pay....but holy cow! I need data on about 500 symbols and that gets pricely with these places. We're talking the price of a nice house at NYSE and CQG. tickdatamarket.com looks better but still pricey, I'll ask for a price quote from them. It seems as though this stuff usually goes to hedge funds and the like. Any options for maybe 2K or less? Maybe, I'll code up a way to get it from IB.
How far back do you want to go? DTN IQFeed has several years of historical 1min data. (back to 2007 I think for stocks), and one month for tick. About $72 per month for US stocks. $300 one off if you want the API. The API is easy to use and overall I have found IQFeed to be very reliable. And it works perfectly under wine on Linux. Because of limitations on downloads, IB historical data is not viable for large numbers of stocks. And you are right, tick data for a large number of stocks is another ball game all together. The storage and processing requirements are quite considerable.
IQFeed may be the best answer. I actually use them as the data feed for my ATS. They are indeed a better option than IB for historical data. I had hoped to get data back about five or more years, but 2.5 years would be useful. I just want to make certain the Pi Trading data isn't leading me astray.
A developer in our group wrote a tick data generato, I can give the program to you for free. If you want to hire him, please contact me. He is Ph.D. from MIT.
I don't know about stocks, but if you want futures data from IQfeed be careful. They only offer continuous contract data and quite a few contracts they use useless rolling rules. Makes the data rather worthless. I have attached a ZG chart below as an example.
I could get "Complete One-Minute Equity Data (OMED) for US Equities" from Tickdata for $48,000. http://www.tickdata.com/html/order.shtml