Underlying: SPX Been there done that <below) 1) TOS stability (via RTD) reliability has become near unusable. 2) TDA-API does not support weeklies, except for some very near term ones. So, looking for other data sources. I have historical, so that is not needed. Note: the snap shot needs to capture all chains of an underlying atomically preferred, but it not possible, each Chain must be captured within a very small time window. -- Pricing must be captured as a set, as the relation between options of a series is critical. Service must support automated data access (no human in the loop).
I've used IB/TWS for such a purpose, offloading 6 expiries' worth, 71 strikes (350 points, inclusive), for ~5 years, without issue. For this purpose, I created a QuoteMonitor page for the single purpose of acting as a database source/repository. From there, I offload onto a .csv, which is imported into a spreadsheet. (The whole process takes ~2 minutes.) I don't usually use their OptionTrader interface, but instead trade from QuoteMonitor pages set up specifically for trading, including a *small* subset of strikes, and then whatever combo lines might be called-for (or *collected* was the week rolls along). Very few pinches on data over the years. The last one -- when 7/8s of TWS was lost for a Friday afternoon, cost my accounts thousands. But that was a first in ~12 years.
I am unfamiliar with IB first-hand. I have heard of issues with extracting large data sets. I need unattended operation. My current implementation captures about 25K quotes at a time (each snap-shot). About 300 Strikes across 20 Expirations (a bit under 12,000 Opras) (out to 90 days) It has been working great for the last couple years, but recently TOS stability has degraded, requiring manual restart of TOS almost daily, which is unacceptable for in-attended operation. Can IB support this kind of load, and is the "snap shot" pricing atomic across all opras (or within a few seconds of each other)? If push comes to shove, I can drop support of the Monday and Wednesday expirations, which currently are not critical to my use, which can reduce the data size substantially, however, that has not been my issue (amount of data).
Can IB/TWS handle your load? I'm not sure -- but here is another thing to mix in -- I pull ~26 fields, for those 71 strikes over 6 expiries. I don't just pull price. (Or -- not just bid; ask; mid; bid volume; ask volume; volume today -- that's 6.... Yeah -- I pull 20 more. I don't know this for a fact, but I *believe* that for TWS, it is the contract that rules the roost, not the fields *for*the* contract. Thus, my "71 strikes" times 26 fields might be just fine, but your 300 strikes (by, maybe 6 fields??) might be a seam-buster. But something to think about. Sorry I don't know more, but if I went 300 strikes and 20 expiries in the SPX, most of that would be zeroes' data anyway. Wish I were more help.
Thnx for the additional detail. Does your setup run unattended? Do you have an API or do you have to run some IB app and poke any buttons, or re-start it from time to time? -- Pardon the "thousand questions"! If I don't find a suitable solution, I may have to redefine my problem! ;-( LiveVol just mentioned they "may" have something in 3rd quarter this year that may be adequate. (Intraday snap-shots)
(You are very welcome.) My set-up is just TWS via Windows7, with the computer entirely closed each night. I am set up to begin to construct a Java-based TWS API, *merely* to scrape data at a predetermined interval, and throw that to a desktop file that will be read in real-time by an OpenOffice spreadsheet. No trading; No portfolio functions besides reading current positions. (So, 2/3rds of the fields-per-strike are market-related, and 1/3rd are portfolio-related.) IB also links/utilizes MetaTrader -- if the current IB API languages (Java, C++, Python and more...) don't fit your wishes..... but you're doing more than what I've done. (And !)