I'll make the general observation that making a robust quote filter is a pain in the ass. I've read all the Olsen paprs on filtering high frequency FX quotes, but that is a very different nature than equities and futures and also only assumes best quotes from particular dealers. When dealing with full order book data you cannot use the same methods. One particularly annoying problem I've had is the non-synchronous updates after the best price changes. When the best prices changes I look at the next level in the book and record that as the new best. The problem is, often this next price is not the best best at all, simply the quote provider hasn't transmitted the new best yet (e.g., removing the best before transmitting the new best). So: best price removed, record new best price, 3ms later the real best price is received and the quote immediately jumps back to its correct level, thus leaving a spurious 2-3ms dip of 1-2% (sometimes worse). I've even resorted to finding the best quotes within 1-second intervals and sometimes there are still outliers. How do you truly define an outlier? You could filter by a movign standard deviation. but in the case of extreme market moves this will reject quotes.. if you dont update the moving stdev then potentially the filter rejects ALL subsequent quotes. Could filter on return between quotes, reject quotes with a relative change above some threshold, this is als susceptible to filtering out actual extreme market moves. Maybe come up with some successive quote filter.. if some quote is extreme, yet the next several quotes confirm the move then the filter accepts and introduces a sudden jump to the filtered quote output. Could also move for relative changes between both bid and ask quotes. It seems that this is uncharted and/or proprietary territory.
You are absolutely right that it is a complete nightmare. I have been working on this problem for two years and expect to spend more time on it in the future. You can cope with jumps by monitoring the statistics of what is being rejected by the filter. If the statistics suddenly change then you can force a re-evaluation of the filter parameters.