Good morning/afternoon/evening I am going crazy as this is the one issue that I still need to figure out before I can go on to make my millions.. haha The situation is this. I have done years and years of meticulous backtesting using IB historical data. At some point I finally realized that there are significant differences between IB's live data and their historical data (I am aware that this has already been addressed in a number of posts). However I continued to backtest using the historical data - this is what I had always done .. my entire system is built around this data. I am now in the process of attempting to automate my strategy using IB's API. It is crucial that the live data I receive is as close as possible to that which shows up when one "refreshes" one's charts to get "historical data", if you see what I mean. Whenever I try to trade manually, I always refresh manually before placing an order, for this reason. Therefore, now that we are trying to code my trading rules, we have looked at the possibility of firing off "historical data requests" at regular intervals in order to mimic what I would be doing if I were trading manually, thus making sure that we are using exactly the same data as that which I have always used for my backtesting. The problem with this, of course, is that not only is this method very ugly and inefficient, but IB have restrictions in place regarding the number of historical data requests that can be made in a given time period. Therefore this method is not at all practical, and I really need to look for a new data feed. What I would like to ask is NOT "which is the best datafeed?" or "which is the most accurate datafeed?" but rather: "which datafeed would give me live data that most closely resembles the data used by IB when they form their historical charts?" Of course, I can try them all out one by one but that will be quite costly. Might it be the case that they are all pretty close so it doesn't matter? Any ideas? I'd really really appreciate any ideas at all! This may sound like a ridiculous question but I hope you can see why I need this. My backtesting has been done, I didn't realize what I was doing at the beginning and it is too late to change now - and the differences can be very significant indeed. There have been so many times in the past that I have lost money on a trade trying to trade off IB's live data feed, only to find that If I had refreshed the charts so that I'd received historical data, there would have been no entry in the first place as the data is totally different! Thank you
You could instead use simulated market data: GBM (source code for Geom. Brownian Motion for simulating stock prices): http://www.elitetrader.com/et/index...-code-in-c-for-generating-time-series.297370/ BSM (source code for Black-Scholes-Merton options pricing calculator): http://www.elitetrader.com/et/index...es-merton-options-pricing-engine-in-c.297419/
The first thing a good system should do is being stable. Being stable means that your results should not depend of coincidental events. From what I read your system is not stable. If you have significant differences depending of which feed you use, it means that you optimized your system for the data. That's a bad sign. Systems optimized for a set of data will only perform well with that specific set of data. Worthless. In a good system parameters, and data, can vary within certain limits without influencing the result. Most quality feeds have more or less the same data, might be off 1 tick every now and then. So different quality feeds should have no major impact on your results. Maybe you should not worry about the data but about your system.
Thanks, good point. I agree. Not at all worried about the system, just don't want to leave anything to chance, no matter how trivial, as I have been absolutely thorough in my preparation with everything else. If we're talking about occasional ticks here and there, I would of course agree that's not an issue at all. Having said that, I really don't think I can use Interactive Brokers' real time aggregated "snapshot" data though as it really does leave out too many highs/lows. I think that would be pushing it a bit. Or do you think even this would balance itself out in the end (win some, lose some)? I wonder..
brynno, I'm in a similar situation as you. I've been backtesting my own systems for a while, using IB historical data, and now i'm implementing my own "algo trader" which will trade through Interactive Brokers API. I think instead of forcing my system to deal with bad-quality data- the responsible and reasonable approach would be to get another data feed from another provider. I have used IQFeed in the past when I developed a project for my client, and have a lot of java code written on top of their socket-based API. I think the extra expense of getting another feed should more than pay for itself by helping to avoid losing trades caused by bad data. Wonder if anyone here has been using IQFeed for real-time and historical data, and how they find the quality compared to Interactive Brokers?
If it's in your budget and you're interested in a purely API-based solution, I'd recommend checking into NxCore. NxCore sends a full feed for any given market for which you've paid (e.g., NBBO and trades for US equities, add Nasdaq Level II for something extra), and its historical data is the same as what you'd get real-time.
Yes, true. Some questions I'm not sure of after reading your post: - If I understand that right some time in the past you "traded off IB's live data" and lost. Did you do this with your backtested strategy (or with another one)? - In other words did you test your strategy for some time with the paper trading account? (Of course the background of my question is that there may be differences in IB historical vs. realtime data but it might not affect performance). - Do you have an idea what exactly causes the problems (e.g. Highs and Lows different in realtime vs. historical or something else)? By the way IB data is better than many people think. But sometimes it needs some extra work...
It's the real time data that is correct. Historical data contains trades that were not made on the exchanges (think funds and brokers trading among themselves). Those trades are reported later but appear in the historical data. Those trades can sometimes be far away from the current bid and ask. They are obviously trades that you can not participate in. One way you can have "correct" data is to collect tick data and make bars of whatever length you wish.