I wouldn't worry at all about minute level data as you're maxed at only 1380 records per day for futures and 390 records for stocks - so no issues there. As for ticks... Each tick record in the database takes up 54 bytes of data. An average Futures contract can get 250-300 ticks/day from IB. This checks out to ~300MB of storage requirements per futures contract per month (~80MB for stocks). By that calculation, you'll need about 10GB per month to store one month's worth of both minute and tick data for 30 symbols. Multiply that by the number of months you want to keep and that's the storage you'll need for market data storage. Multiply that again by ~2 and that's your server requirements
Straightforward reasoning there, thanks! However, in the case of tick data from a provider other than IB, seems the tick count will be much higher, esp in the case of a market like NYMEX Crude Oil futures. On a separate note, while researching the db side, a blog post concerning IB's API in a Docker image came up - have a look, might be of use to you in that area: http://ryankennedy.io/interactive-brokers-api-docker/
If you're using a tick-by-tick data feed then you're going to need more disk space... IB sends aggregated ticks 3-5 times/second. Most people, however, don't really use/need tick data and should be find with second-level data. It may be a good idea to add an option to QTPyLib that tells it to store second-level instead of tick-level data in order to save space. What do you think? I haven't played with the Docker image you mentioned, but it looks like it should do the trick (though I'm not a huge fan of Docker myself)
Hmm, yeah for sake of space reduction the 1-sec bars could prove useful. Personally I haven't used Docker, just read about its benefits (and the opinions of those in the programming arena who see it as over-applied nowadays). Figured it's good to have options, so that package adds one to the periphery if the need arises.
Hey ranpo, looks like you're keeping a regular pace to the updates & improvements to QtPyLib - inspiring to see! Regarding SMS, have you messed with pygooglevoice at all? I tried a sample script as-modded from some code posted to stackexchange & a blog, but couldn't get it to fire. Would be cool if a free option like this could be integrated with a platform as powerful as yours.
I'm not based in US and don't have Google voice here - so there's no way for me to develop and test such integration at this time... I've opted for Nexmo/Twilio due to their international coverage and very reasonable rates (I'm using Nexmo for almost a year now and I'm still living off of my initial $20 deposit )
Ran, does the backtesting feature in QTPyLib work off of its own logic/code for opening & closing positions from the historical data array chosen for testing? I'm not aware of any backtesting features built into TWS, let alone anything accessible through their API. Do you make provision for use of limit orders within this module?
Yes to all QTPyLib has it's own backtesting logic. If you use market orders in backtesting, it considers the next tick/bar open as the trade price. If you're using limit orders, it considers the limit price as the trade price only if the next bar range touches your limit price. For obvious reasons (and this isn't limited to QTPyLib), both methods not 100% representative to real-world trading, as I'm not aware of the order queue and other market structure variances that may affect the actual trade.
Hello Ran, I watched your recent webinar, thanks for making time to do those! Got two questions which I posted to the input box, but moderator didn't pick them up: 1) If you were looking at on_tick subscriptions for 35 instruments at a time, would QTPyLib be capable of handling the bandwidth and, if so, would a different data storage choice be more appropriate (e.g PostgreSQL), instead of HDF5? 2) Have you considered (not necessarily planning) adding another market data connector (e.g. IQ Feed) or broker via another platform like Rithmic, CQG, etc?
Thanks for attending the webinar I had a great time doing it. To answer your questions: 1. Starting with version 1.5.56, QTPyLib is fully multi-threaded. It was tested with the entire S&P500 constituants and it worked without problems on a 8-core machine, although the total number of threads it can handle is machine-dependent. When running your strategy, use: Code: $ python strategy.py --max_threads [NUMBER] ...and play with the [NUMBER] part to find the best settings for your machine. That being said, I don't foresee a problem handling 35 instruments on a decent machine. 2. I am toying with the idea of adding more market data sources, but it'll require me to do some serious thought about possible issues that may arise (for example: I'll need to find an elegant way of mapping the symbols of each data source to IB's format to keep the simple syntax). But the short answer is YES - this is something I'm planing on having added to QTPyLib someday sooner than later. Cheers, Ran