Works like a charm !! Awesome !! I used this to modify the original program.py... many more questions on the way !! Thanks again !
Have fun! Although debugging and integrating error handling stuff is a nightmare to handle in TWS API async framework.
That's absolutely true, it has slowed me down significantly. The next big challenge: getting over the pacing violations, now I have 150 Instruments for which I need 11 bar sizes each, for a period of 5 years. So given the pacing violations: Making identical historical data requests within 15 seconds. Making six or more historical data requests for the same Contract, Exchange and Tick Type within two seconds. Making more than 60 requests within any ten minute period. How do I plan this, doing this manually will be a nightmare, considering all contracts are defined and resolved, how do we use the timer function iteratively. In addition to this I will need to download the High resolution historical ticks with the following limitation: 1000 ticks per request, which is roughly 5 mins of historical ticks (given that there are more than one ticks per second). Thanks in advance. I am learning a lot!
Seems more like a side project than a plan to actually build some proper data infrastructure? Sounds like you could save weeks of work (at least) by using something like IQfeed.
I see the flaw in this approach, specially the amount of time/Effort it takes, what would be your suggestion for a proper data infrastructure given that I would need APAC exchanges such as NSE and BSE (level II/III) as well, for which there are barely any vendors coverage. Thanks !!
Luckily you only need to download all this historical data once. Once you have it on your computer you can add newer data to it, resulting in less requests to IB's data servers. My guess is that you are not in a hurry to receive the old data. If you were in a hurry, you would have started this work multiple months ago. So you need to write software to submit only 60 requests per batch. Once the batch has been completed you wait until the 10 minute period has finished before you submit the next batch of 60 requests.
Indeed, a whole other set of problem. And I don't think that IB provides any support on this topic, so another source of data may be necessary.