Craig & winnertakesall, you're right about the orderState messages and I should clarify. When I submit an order, I do get the orderState and orderStatus messages and those work but the problem I'm having is when I *update* an order. The way my algo works is as follows: 1) It looks for specific price action and when found, a LMT order is sent if the desired price is too high/low or a MKT order is sent if the price is even better than what was desired. 2) As soon as some of the order is filled, 100 out of 1000 shares for example, target LMT and STP orders are sent with the corresponding filled number. 3) As an order continues to fill, the target and STP orders are updated to match the position size. This is where my problem lies. There are no update acknowledgements from IB and if you send updates very quickly, one after another, sometimes the orders don't get updated.
EliteTraderNYC, yeah, I agree. I'm well aware of IB's data shortcomings and I will definitely be transitioning to another provider when the software is ready. For the time being, my strategy is so simple that data accuracy isn't as critical as one might imagine but I've got a few ideas that will require a solid feed in the future.
Also, from the IB api mailinglist, I've learned that IB is sort of aggregating data over a 'several' (100+?) milliseconds and sends that out. So multiple trades are packed together and returned as one trade. That's really ok, but if in between those separate trades a change in the dom happened (ask/bid size+levels) I would not see that related to a particular trade. It's likely those are accumulated as well. My algo is looking at the dom and at what levels a trade took place, so losing (too much of) that data would likely degrade the analysis. On the other hand, I have no idea how many actual separate trades take place on the ES mini futures on a daily basis average. Does anyone have good numbers on that? I've read that it's possible to get more accurate data for an extra data subscription. Has anyone tried that, and is the difference significant?
All lot of retail bd's conflate data but it's is not exactly what you described. Conflation is when they send you data every say 100 milliseconds. So say 50 prints happen in that period of time, you get all 50 prints, but you don't get them until the data is distributed in 100 milli intervals. Not good for low latency trading, but the data is completely accurate.
Interesting thread as I have been down this road myself. Have your investigations yielded the reason as to the discrepancy? Is it a latency issue? From personal experience I can recommend both the Rithmic and CQG APIs as being significantly superior to the IB offering.
Ok, so what you're saying is that the ib api client does indeed see every single trade and dom change, but not exactly as it happens but instead at bursts interval? If that's the case, then this would be a lot better than what I was assuming. Extreme low latency trading is not something I'm competing in, I'll leave that to others ;-)
Thug, the discrepancy was due to a bug in the software. I was using the wrong variable when calculating the price at which a MKT order would have filled. Interestingly enough, a rough calculation of of 0.1% slippage has kept my backtesting results within about 5% of live trading. I could certainly improve the model but that's pretty low on the priority list at the moment.
I figure it's time for another update after running live for two months. Progress has been slow now that we're back into golf season but I'm still moving forward. May was a good month with a win rate of 70% but I'm still getting trade irregularities when compared with back testing results. Clock synchronization has certainly helped but the problem is now directly related to the IB data feed I'm using. Fortunately, the discrepancies have been mostly in my favour but I am well aware that this can only be chalked up to luck so I will need to transition to a new data provider. I thought I might provide a few more details about the system: The GUI is a simple command prompt style interface.; Orders appear in the IB TWS software and can be manually manipulated.; The entire system is event driven with only a few small exceptions run in separate threads.; Total code line count is approximately 12,000 lines over 82 classes.; I use Eclipse on a MacBook Pro as my development environment.; My server: Intel i5 4570, ASUS H87M-PLUS, 8GB DDR3-1600, 2x500GB in RAID1, runs Windows 7 Pro, and I use Remote Desktop for remote administration.; and I made a small Arduino device with LEDs to serve as system indicators (Power, Internet, Open Orders, Open Positions, ERROR).
IB is not true tick feed but rather "delta" feed to update quote board and reports changes in bid ask and separately volume changes in intervals of about 300 ms. It ensures fluid interface for all customers regardless how fast markets move. Real time bars consolidate trades into bars over 5s and are reported at the end of 5s period. For DOM data there is no guarantee that all prices will be displayed - again IB is more concerned about the application not freezing than delivering all possible information.
I think that IB API specifies that it is up to the programmer to keep track of changes to order quantity. Check if you are not exceeding allowed number of messages per second sent to TWS. This may explain why orders are not updated. You should receive error message in that case.