This is good to know. There seems to be a gap in JDSU 1/7/2008 into February and some other spots. I guess any meaningful analysis would need to compare to other data. Like I wrote before, I'm very pleased overall with the data. My analysis is kind of meaningless since the minutely gaps don't mean anything about the quality. It makes me feel better to have looked at things more closely - though
Also - Sorry to Pi Trading and anyone who wasted time with my sheet. I'm glad to know the data is in good shape and I'm impressed by the reasonable price. A silly mistake to think every day should have 390 bars. If there's any other analysis that would be worthwhile, let me know any ideas.
You say that IB is throttling the data? IB still has most expired contracts going back a couple of years or so. I can download each contract in maybe 10 minutes give or take while I do something else. In Multicharts you can then edit the data, combine it, convert it to ascii etc. You can download a free one month trial of Multicharts. You can do the same thing with Sierrachart, then export the data to Excel to work with it. How long does it take you to download a contract from IB?
I didn't try Multicharts - probably depends on if I could automate the ascii dumps of the entire SP500. I was using the Excel interface to IB with some homebrew VBA automation to do this. What I was doing was working but couldn't do more than 60 requests per 10 minutes. Given the limit on the time span for minute data requests, I could see it would take weeks to get the amount of data I wanted. I may still do this to fill in any gaps and to keep up to date.
I just read quickly through your question and assumed you just wanted something like the ES futures for the last year, not all 500 stocks from the S&P 500. With Multicharts it takes about 10 minutes to download about a year's worth of data for each stock from IB. If you only downloaded one stock at a time, it would take about a week to download 500 of them. Then of course you're probably better off with some of the suggestions here, sign up for a trial with a different data provider or buy a DVD of data. Then again you might do well just to continue with your visual basic automation if it will download the 500 equities in a few days. Too bad there wasn't a place here on Elite or on the net where we could share files of data. I wonder if we're allowed to share the files of data that we collect from IB? Is there something that we sign or such that doesn't allow sharing?