how the hell do you guys manage to start a flame war with each other in a thread discussing historical data feeds? the mind....boggles.... All those vendors are making this way too complicated, one site wants me to use his software just to read the data, another has it in some odd format that i need to use a lookup table to debug. WTF? Anyway i found a good site for es intra tick data by searching et: anfutures.com Those guys kept things simple and the contracts are already continuous with date/time/price, that's it. So you just download about 10 GBs of data in csv file and do whatever you want with it. I already have it all loaded in the database, about 8 million rows per contract period, most are duplicate as it's tick data. Distinct data is about 2 million rows per contract. Did some initial verification using random samples and database group by, data looks pretty good.
thanks for the website where and how do you store the data? how long does it take you to backtest anything with this amounts of data?
in mySQL database, look into the "LOAD DATA INFILE" command for batch data uploads. I havent done a complete run through backtesting yet, as i just finished loading the data(took 12 hours on my local core2duo desktop) and starting to write the backtesting system in java. In term of performance, it largely depends on how intensive/complex your strategies are for each iteration. For me, as i said i am building a point & figure engine for my backtesting, so volume is of little consequence, meaning i just need the distinct row for each date/time/price. After all the data massaging, it's about 24million rows total. I put it all in 1 table with a clustered primary key on date+time. The performance is excellent, mysql database has really come a long way since the early days. Anyway for java, i will probably wont go too crazy on multi-threading, just use 1 thread per strategy. So within each iteration, all strategies can get executed in "parallel".
for free???? I have no idea if they do, but I would bet that the old maxim applies: you get what you pay for.