IMO the TS Data Integrity department lacks integrity. https://www.tradestation.com/Discussions/Topic.aspx?&Topic_ID=62714 https://www.tradestation.com/Discussions/Topic.aspx?Topic_ID=62716
I've never read that book. But from my limited understanding of information theory and signal processing, my intuition is telling me that the only way to strip out bad ticks is to make an assumption about what price movement 'should' look like, thus by definition invalidating the modified historical data, near guaranteeing deviations (whether filtered data or data with bad ticks gives higher deviations, would depend on the system). I think the only way to get clean data is to start with it, anything else is just guess work and assumptions on what data 'should' look like. Is clean data so hard to get that it is an unreasonable goal to request? I have a feeling that 99.9999% of the data coming through my L2 is clean, though in 'ridiculously' heavy market activity things mess up a bit. Those points you would need records from the exchange to get a good picture of what happened.
Both of the threads in the links above expose problems with bad historical data in TS. One of the threads contains code to find bad data points. What is shown is that there's dozens and dozens of stocks that have wrong price values and some of these discrepancies go on for months. A couple different issues are being discussed with regard to bad data. One is bad historical data and the other is bad data points in real time data. In my mind these are separate issues that are addressed differently. If historical data contains serious flaws the confidence level of our backtesting will be seriously compromised. In the case of serious flaws in the historical data the backtesting should be considered unreliable and the symbol discarded until reliable data has been obtained. In real time, we may see ticks that are at a later time corrected by TS data integrity. The result is that an auto-trading strategy running live may place orders based on the live data and later changes may be made to the data by data integrity resulting in backtesting that shows no trade at the point on the chart where an actual trade was placed in real time. Unfortunately I know of no way to reconcile this type of occurrence. TS states on their web site: Exchanges send data to TradeStation that sometime do not coincide with current trading activity. TradeStation tries to filter this data to remove trades of this type, so that such trades, which may be misleading, do not alter your Quotes or Hot List windows. For example, an "out of sequence" trade may come in that indicates a price higher or lower than the price at which the stock is actually trading. Because this trade appears to be "out of sequence," the time at which this trade was actually placed is probably unknown and, therefore, TradeStation removes it from the data. This trade is still valid, however, so it will still appear in the Time & Sales window and be flagged as "out of sequence." If we choose to write our own code to filter bad ticks that are above or below a threshold we define as a bad tick; this filtering needs to be capable of identifying ticks during highly volatile periods as valid ticks. And if such filtering is incorporated in real time trading, it must also be incorporated into our strategies for use while running our backtesting and walk forward analysis.
I could only go by the tone of your OP along with the lack of any mention that you'd already been in contact with them. IMHO, the whole business with bad, missing or out-of-sequence ticks is part of the cost of doing business. In the larger scheme of things that's much less of a worry to me than data that's lagging the inside market, power loss and network outages. You do your best to minimize their impact since there's no way you can eliminate the chance of them occurring.
Wouldn't tracking the current bid/ask and using that as the source to build historical data fix this issue? Its a fairly major issue for me since bad ticks cause deviations from my simulated system multiple times per day. It would seem to prevent the issue of lagged trades. And I am sorry if the tone of my post was 'bitchy', in no sense was it written that way, I hold no ill will towards TS, but am ready to move on to a data provider which doesn't have this issue.
I'm curious why TS allows trades which are so obviously outside of the real time bid ask spread to affect their data feed, and also get saved to their historical data, invalidating backtesting. Seems like it would be easy to limit the acceptable trades to the current NBBO. Seems like basing data on the NBBO would be the best solution.
Short of cleaning ticks in real time, which I'd personally not want done, how can any data provider guarantee the data you receive is clean when they're not the source for the individual trades? Best case, the most they can realistically do is give you a service level regarding maximum lag time and, preferably, an alarm once that delay reaches a given threshold and they can guarantee the cleanliness/completeness of their historical data. But, as another poster points out, do you really want that since clean/complete data should not be an expectation when live trading. With that in mind I'd expect that a top tier data provider would have to provide you with two data sets, one is the data stream that was sent live and the other a reconciliation set for the adds/mods/dels. As far as the NBBO goes, that's an asynchronous data stream to the tick stream. At best you can use it to develop a statistical model but, that's got its own host of exceptions that you're going to have to account for to make it usable.
Our tick filtering mechanism allows trades that fall slightly below the inside bid or above the inside ask. Weâre in the process of moderately tightening this filter moving forward, and we have removed the questionable data that was reported to the tape on 11/19. Youâll now see the updated data if you go to View â Refresh â Reload or use CTRL + R. For any follow-up questions please post to the TradeStation Support Forums. Sincerely, The TradeStation Team
The sad fact is that there's simply no way to tell if an outlier that's well outside the NBBO was the result of a glitch somewhere along the data path, a "fat-finger" trade done by a human or an algo that's run amok.