Yes, I agree. That wasn't included because I consider it a given. Both of the approaches to including the missing data avoid adding significant numbers of extra packets or extra data in the same packets. The timeliness of data is very important but the overheads of TCP/IP protocols dwarf the extra size involved in adding one more number to the packet.
I realized later that I should have used the word "significant", too, but I thought that was a given also
That's well explained, and gets my vote ! By the way: it is not only the third-party software who needs this, it is needed for TWS's own charting as well.... If IB takes their own charting-tools seriously (and I'm sure they do), then it is definitely needed. The real-time-charts will only show the correct highs/lows per bar when you do a backfill-request (at start up), but there after the charts are filled with the real-time snapshot feed, and will miss quite a lot of highs/lows.. When programmed well (at the IB-server-side where the snapshots are taken), it does not need to give much overhead.
Having changed to IB in the last 6 months I really appreciate the stability as compared to some other vendors with every tick data feeds, where I often experienced delays and timeouts etc. So your suggestion to get the additional info and mantain the stability is great and gets my vote. Thanks for starting this thread and I sure hope IB will act on this. Regards Fraser
Is there any chance that we can get an opinion from one of the IB ( Interactive Brokers ) people who post on this board?
Someone challenged me to prove that the data was doubtful. I was curious about how bad it might be so instead of using "other data" I used IB's own backfill. According to ane earlier post here when you get IB backfill you get every price they know of but when you get real time they skip all but the last price in each 200ms period. So what happened? The upper price series is the real time ib series (30second bars) The middle is collected by backfill (should include all prices) The bottom shows red wherever the ohl or c varies See the pictures:
Remember the TWS data stream is not time stamped, so bar boundaries are determined from the PC clock. If the latter is off, then your bars boundaries will be off too. If the PC clock is not synced frequently via NTP, it will certainly drift - by quite a lot. Add in internet propagation delay and local PC processing time. You will never get short period bars to match exactly. I've been through this exercise some time ago, and I found the match up was quite good over the few samples I tested. By frequent clock synchronization I mean something like every 30 mins or so. So what I am saying is that some of the differences are not due to missing ticks.
Since you asked... This isn't my area of expertise so I'm not going to touch this but will note that a few people who might be willing to share their thoughts are on vacation at the moment.
Good point. Onto that - I had a drift problem a few months back so I run Dimension 4 and it synchs every 15 minutes. The correction during that period never exceeded 15ms. I'll run one when the afternoon session starts with 1 minute time corrections off the HK govt timeserver (I normally use a close Telstra timer thats slaved off their internal time server). Edit: And thanks to def. Would appreciate their input when they get back. IM(H)O it would be more important for IB to improve the data quality by one of the measures suggested here than to, say, make the next couple of improvements to IB charts with "weak" data.