Sorry if this sort of thing has been asked and answered before, I used the search function and came up blank. I'm a computer scientist in real life and I'm trying to leverage my own expertise in machine learning to make my trading more intelligent. So I'm trying to find a retail broker that has a strong historical data offering and actually lets me use it in my own applications. This would be my first time using a broker where I have to pay for a data feed, so there are some aspects I'm not clear on. I'd prefer to not have to purchase a separate data feed (like IQfeed) from the broker's own, but if I can replace the broker's and not have to pay for the broker's, that's fine too. I don't need level 2 data, level 1 is fine for my purposes. I took a look at the offerings on the IB website and while the API itself is quite usable, with the historical data download limitations they are simply a non-starter. Maybe there is another way I can use them though and not have to go through their API (like with a third-party feed)? I called Tradestation to ask them about their offering (it's something they tout on their website) but they weren't able to tell me if I could actually export en mass their historical data (years of minute bars or months of ticks) for symbols into a CSV or Excel file. I don't want to try to implement a set of machine learning algorithms in EasyLanguage, I want to take the data and put it into things I've already developed! Similarly I called Lightspeed and asked the same question and they said I might be able to do it with RealTick, but I'm not willing to buy in on a "might" and I can't find any information on their website about historical data. Also the monthly cost of RealTick with them is a tough pill to swallow when I won't even be using most of its features, but if that's what it takes so be it. If any of these platforms had demos WITH DATA so I could test I'd be happy to do so and open an account, but that seems to be a tall order. Has anyone else been in the same boat I am? It's been difficult to wrap my head around some of the offerings out there and the sales reps you hit by the phone lines seem to be mostly clueless about this sort of thing because it doesn't seem to be a common request. At least they admit they're clueless about it. Anyone else look for a similar offering and find something? edit: Just to be clear, I'm interested in equities/options data, and data for futures on indices like SPX.
Yahoo provides free 5-minutes intra-day data for upto the past 15 days. Google also provides free intra-day data for 10 days if I remember correctly. They are easy to download by wget or lynx or anything you like, and translate into csv format to feet into your program. For real-time Level 1 data, I believe you get for free from most of the brokers if you have an active brokerage account with them. As for historical EOD daily data, Yahoo's eod data for many years for a lot of symbols, you get for free.
Maybe I wasn't completely clear on what I'm looking for. I appreciate the suggestion, but EOD data is not what I'm looking for. I need much more than 15 days of intraday data: I would need to cover multiple years at five minutes, at least back to the prior market conditions in 2007. Historical EOD data is fairly straightforward to find, though the free stuff isn't particularly good. It's intraday that I need.
You're asking for different types of data. While IB limits data requests, it's perfectly fine for equities but yes, you need patience, time and planning as it will take longer. I've dealt with timeframes as short as 15 seconds with IB API. Not sure where you could get expired options data anywhere for free.
IQfeed offers a free demo and it costs $50 a month...that's probably as good as you're going to do. You're not likely to find a free data provider that let's you bulk export reasonably formatted data across asset classes without a subscription fee or expectation of commissions.
www.qcollector.com and a months subscription to iqfeed or esignal on demand will get you all the data you require.
I will second the qCollector recommendation for equities and futures- I have not touched options data so I can not comment on that... The developer is fairly quick to respond to any questions, which is always helpful.
If you wanna play, you gotta pay : NxCore. A bit expensive, but high quality data. Unfortunately there are no free rides
Thanks for all the suggestions. Just to be clear, I'm quite willing to pay and I expect to do so. What I didn't want to do is be forced to pay for a broker's data feed and then have to also pay for a third-party data feed which supersedes it completely. This post was originally in the broker forum and got moved so perhaps that's where the confusion is coming from. In retrospect that's more of a question for the broker's sales rep and that's exactly where I'll take it. Also that pointer to qcollector is fabulous, thank you! Especially since I already took a look at iqfeed and liked their offering.