JDSU minute data backfill: (Chart made with DTN.IQ software: ) Note the 15min bar containing a downward spike which makes the data useless for P&F charting.
Historical minute data for futures is also filled with spikes. It would be totally useless for bactesting. I don't use it for backtesting, but for backfill up to 5-10 days back and I have to erase the spikes individually with my charting program each time I have to backfill. You would think they could put in their own spike filters... ?
I use DTNIQfeed for charting ES, and I don't get any spikes with historical 1 minute data..... And in fact, their feed seems to keep up MUCH better during times of intense trading activity. I haven't seen any lag in quite some time....
Thanks for being open and sharing your experience wsws. Those are some long tails! I was looking for some current info on these guys as a possible feed for Ami or QT. I hope iqfeed will post an update in this thread. I say this as someone seriously considering their service.
Using IQFeed's historical time&sales tick data, I investigated some of the spikes shown in the chart examples posted previously: Conclusions: (1) If the spikes have really happened as mistrades, it is actually impressive that IQFeed delivers all ticks including the mistrades (even though that's not what the user wants when backfilling historical minute bars instead of historical tick data). (2) For the download of historical tick data it would be a good and viable approach not to filter bad ticks out but to simply mark them as questionable, as already suggested. Reason: The processing of downloaded ticks is done by the client application, which can freely decide what to do with questionable ticks. (3) For the backfill of historical minute bars calculated by aggregating tick data, with the tick aggregation conducted by the IQFeed servers, it leads to the unwanted spikes in the chart when the tick aggregation also includes ticks that are known to be bad ticks. How can bad ticks be safely detected? According to iqfeed (Jay F.), only minute backfill data is formed by aggregation of tick data. Daily data, on the other hand, is NOT aggregated from tick data. If at the end of the trading day, the IQFeed quote servers have an end-of-day bar (daily bar) for the trading day, which can be considered authoritative (as it is not formed by tick aggregation), a simple solution to safely detect bad ticks is as follows: Every trade (tick with volume >0) that occurred above the high or below the low of the end-of-day quote must have been a mistrade and hence got busted, otherwise it would have been reflected by the high/low range of the end-of-day bar. This should safely eliminate the majority of spikes, with the filtering being correct in the sense that no valid trade got filtered out. The worst and most disturbing spikes would be safely eliminated, greatly improving the quality of the backfill minute data. However, the filtering approach would not be complete, as it cannot detect mistrades that occured within the high-low range of the end-of-day bar. An interesting question is here whether the data feed vendors get notified by the exchanges after trades got busted, which would allow the data feed vendors to filter out all bad ticks at the end of the trading day. It would be a great improvement if DTN could implement the proposed filtering approach for the generation of backfill minute data. JDSU: probably a mistake during order entry: 0.34 instead of 4.34 low of authoritative EoD bar for 03/12/04 is 4.25 => trade must have been busted (as it is not reflected by EoD bar) and can be filtered out at the end of the trading day 03/12/04,11:55,4.339,100,4.33,4.34,21591883,,,Trade, 03/12/04,11:55,4.34,800,4.33,4.34<<,21591783,,,Trade, 03/12/04,11:55,4.33,500,4.33<<,4.34,21590983,,,Trade, 03/12/04,11:55,0.34,300,4.33,4.34,21590483,,,Trade, spike! 03/12/04,11:55,4.33,200,4.33<<,4.34,21590183,,,Trade, 03/12/04,11:55,4.34,200,4.33,4.34<<,21589983,,,Trade, 03/12/04,11:55,4.34,100,4.33,4.34<<,21589783,,,Trade, INTC: An example for a couple of trades below current bid price. low of authoritative EoD bar for 03/08/04 is 27.62 => as trades are below this low, they must have been busted and can be filtered out at the end of the trading day 03/08/04,14:06,27.9700,1700,27.9700<<,27.9800,61070682,,,Trade, 03/08/04,14:06,27.9700,250,27.9700<<,27.9800,61068982,,,Trade, 03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068732,,,Trade, 03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068532,,,Trade, 03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61068332,,,Trade, 03/08/04,14:06,27.9700,1500,27.9700<<,27.9800,61068132,,,Trade, 03/08/04,14:06,27.9700,100,27.9700<<,27.9800,61066632,,,Trade, 03/08/04,14:06,26.9600,161,27.9700,27.9800,61066532,,,Trade, spike! 03/08/04,14:06,26.9600,200,27.9700,27.9800,61066371,,,Trade, spike! 03/08/04,14:06,26.9600,202,27.9700,27.9800,61066171,,,Trade, spike! 03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61065969,,,Trade, 03/08/04,14:06,27.9700,200,27.9700<<,27.9800,61065769,,,Trade, 03/08/04,14:06,27.9700,100,27.9700<<,27.9800,61065569,,,Trade, 03/08/04,14:06,27.9700,1300,27.9700<<,27.9800,61065469,,,Trade, 03/08/04,14:06,27.9700,301,27.9700<<,27.9800,61064169,,,Trade, 03/08/04,14:06,27.9700,501,27.9700<<,27.9800,61063868,,,Trade, 03/08/04,14:06,26.9600,100,27.9700,27.9800,61063367,,,Trade, spike! 03/08/04,14:06,27.9700,130,27.9700<<,27.9800,61063267,,,Trade, KLAC: Trade @ 54.46 above high of authoritative EoD bar (54.03) => trade must have been busted and can be filtered out at the end of the trading day 03/05/04,12:11,53.4300,400,53.4200,53.4300<<,6771260,,,Trade, 03/05/04,12:11,53.4300,100,53.4300<<,53.4500,6770860,,,Trade, 03/05/04,12:11,54.4600,10000,53.4300,53.4600,6770760,,,Trade, spike! 03/05/04,12:11,53.4400,100,53.4300,53.4700,6760760,,,Trade, 03/05/04,12:11,53.4500,400,53.4400,53.4700,6760660,,,Trade, again a trade above high of authoritative EoD bar (54.2) for 03/08/04 => trade must have been busted and can be filtered out 03/08/04,12:51,53.3000,100,53.3000<<,53.3200,3606995,,,Trade, 03/08/04,12:51,53.3200,100,53.3100,53.3200<<,3606895,,,Trade, 03/08/04,12:51,53.3200,400,53.3100,53.3200<<,3606795,,,Trade, 03/08/04,12:51,56.2500,10000,53.3100,53.3300,3606395,,,Trade, spike! 03/08/04,12:51,53.3300,122,53.3000,53.3300<<,3596395,,,Trade, 03/08/04,12:51,53.3300,100,53.3000,53.3300<<,3596273,,,Trade,
example: e-cbot Error Trade Policy http://www.cbot.com/cbot/docs/45972.pdf November 6, 2003, Effective November 24, 2003 3 Trade Price Outside of the "No Bust Range" If the price of the asserted error trade is more than the specified number of ticks from the reference price, e-cbot Operations will send a broadcast message to the user community indicating that the trade has been called into question. If the asserted error trade is outside of the specified tick range and involves only two parties, e-cbot Operations will attempt to contact the parties to the transaction. If both parties agree to bust or re-price the transaction, e-cbot Operations shall send a broadcast message to the user community and an alert to the quote vendor network indicating that the trade was busted or re-priced. Do other exchanges send such kind of alerts to the quote vendor network, too?