If you are relying on anybody's tick data to reconstruct bars then you are fooling yourself. I have used Bloomberg, IB, and several other data providers. They all have issues. IB takes the best approach in that they provider 300 ms bars as that tends to filter the outlyer data. And if you can deal with 5 sec intervals you trade their 5 second interval bars, which reconstruct perfectly to the historical.
Depends on the timeframe you're trading. And that depends on the cost of lower latency. And that determines whether it's worth capturing the spread at all. If you're fast, your strategy can be simpler & more passive.
Well I have constructed 5min and 1min bars this weekend for 600+ stocks and found that I have a 97% match between IB TWS bars and those provided by my charting platform eSignal. So for stocks NYSE/NASDAQ the data is good enough. I wish people on this forum would give actual real statical data rather than just rant this is bad this is good e.t.c. Give us some facts so that we know you have actually done back testing or do actually use these platforms or do have a programming background. The HFT guys obviously cant use IB TWS as you need a platform that actually resides close to the mkt because of data latency. And anyone who says back testing is a waste of time, how the hell do u test your code with real $$$ if so I know your talking BS. Thanks James
You can demo Mirus /w Zenfire and sign up to the Mirus demo server with Ninjatrader. Then translate your algo to ninjascript and use Ninjatrader 7 to backtest for free.
* I have coded against IB, as well as directly to an exchange or two. Currently, HFT is not so much about data latency as network latency. Location is about physical network length & # devices. That's how perverse it has become.. Any retail trader who thinks they can compete by running similar strategies from home is .. wrong. As for IB, I see about 500ms between sending an order, and receiving an ack. That's a lot. And, considering that delay, and adding whatever their data latency delay is, I think we're talking up to 1 second in latency. So .. that moves IB based strategies away from HF, and toward something else.
Given the history goes back to 2000 for many instruments, this is more history than most popular data providers offer. Do you know if there is an easy way to extract this data from files provided for use outside Trading Simulator platform?
Thanks for the pointer! I will give it a try. Maybe there is a mess-up in my charting software with contract codes, such as using 1-digit year instead of 2 digits... still works fine for the current month. Edit: I just checked it and my favourite charting package is indeed able to extract data for expired futures contracts from IB when contracts are specified individually. Perhaps, there is a problem with requesting data for expired constracts in the "continuous contract" algorithm. Thanks JackR!
I agree there is no "ideal world". In particular, in an attempt to privide higher-quality historical data many providers "scrub" both historical "tick" and "bar" data in addition to filtering they do in real time. The difference is just in how often there is a discrepancy... and with IB the problem is more severe than witha few other data providers: 1) IB real-time data is "sliced". The last trade price is sent 3 times a second or so as opposed to every tick. So, bars constructed from real time data have incorrect high and lows very often. 2) IB real-time data lack time stamp. As result, beginning/end of bar depends on how long it takes the data make way from IB to the trader's computer and on the trader's computer clock. So, open and close of a bar is off on most bars during periods of higher trading activity. IB made a reasonable trade-off to make sure they can deliver data that doesn't lag during spikes of market activity without investing in infrastructure required to deliver reliably every tick. Given the price you can't really complain.
My belief is - If you are going to trade that data stream, test against that data stream (as best as can be done). Of course a lot depends on what type of trading you are doing. Jack
When forward-testing, yes absolutely. However, it's worth being aware that historical data from most data providers is different from their own real-time data.