I am not touching futures at the moment ,after the generous recommendations I got from a thread I started last week in the future forum section, I will execute long and short on SPX/SPY scaling up as much as i feel comfortable at this stage, few people that use futures need to execute transactions every day as they trade for a living and I am not in this position so far, so can wait solid setup come to me, even manually with scanners Regarding the data I am gonna go back from the beginning, because one of my strategies have to take into account all the historical data. I am thinking to get both the hourly and daily although I could derive the daily from the hour with some algo I guess, but do not want to mess up making errors calculating the gaps in the wrong way. Just eating the elephant, need to do a piece per time, no rush, can do trading and be profitable also manually if I select them without stupid over. Just do not know if can get also UNLISTED stock with IBKR to execute better unbiased backtesting
That sounds like a wise decision to me. Downloading both, daily and hourly, is indeed simpler albeit that it will take longer than if you were to calculate your own daily bars. Be aware though that there is always a chance that a daily bar has different data than the hourly bars of that day. No database is perfect, not even at IB, so there might be discrepancies. I am not familiar with UNLISTED stock.
I need now to build in practice my db, excluding time.sleep that is of course is a bad practice, I am wondering what is the best practice to retrieve thousands of different historical tickets for what concern limit requests from TWS API. Assuming you in the forum are using Python, Is in these cases used ib_async or a custom implementation with `historicDataEnd()`? I am scared I could have inconsistencies and lose money because of wrong backtesting data consistency due to the TWS API limitations
Please refer to note 1 Pacing violations from https://interactivebrokers.github.io/tws-api/historical_limitations.html Requesting too much historical data can lead to throttling and eventual disconnect of the API client. If a request requires more than several minutes to return data, it would be best to cancel the request using theIBApi.EClient.cancelHistoricalDatafunction. To make my first database I need one time daily and (maybe) hourly, going back to '87. Being disconnected and/or getting inconsistent data would be frustrating and dangerous First of all I would like to find a "resume" mechanism or a way to download data in batches, so that can manage gradually a couple of hours per day, I should implement some logs to insure i downloaded all the nasdaq universe(ex. counting total tickers) and on top of that I do not know if should use ib_async, or writing all the logic myself manage the concurrency, calls to be sure that are executed, or if IB().sleep (instead of time.sleep()) could be an easy way to get around. Finally I do not like the idea they are going to slow down my connection, nor that i have to use on top of IBKR another cheap/reliable data provider just to get historical data, because if I subscribe to IBKR I expect i can download the historical data for the NASDAQ universe, I am not asking for 1 min candles.
The pacing limitation means that you can submit maximum 50 reqHistoricalData() requests per second. For your application, where you want to download "thousands" of requests, that would not really cause a limitation at all. Getting all that data from IB will take time anyway. Simply put a 20 millisecond delay timer between each request you submit and you have covered this limitation. Note: this limitation does not apply to the data which IB sends back to you. It only refers to the messages you send to IB.