I wonder how to consolidate HFT raw tick data into a more standard set with their respective volumes obviously. Any idea, suggestions or hints would be appreciated Thanks!
Your question is unclear: What do you mean by consolidate? What is "HFT raw tick data"? Are you talking about trades, quotes, something else? What is your data source? What is a "standard set"? My best guess is - are you asking how to compute OHLCV bars/candlesticks from tick level trade data?
By a "Standard" I meant "regular available tick data to retail clients". Let's put a simple example, forex tick data. As you know Interactivebrokers gives you the option to receive it in 1/10 pips or 1/2 pips. Any tick has its price, volume, Bid-Ask sizes. So, having said that, let's say that you only receive forex in 1/10 pips but you find it's better for you to have it in 1/2 pips. What method or algorithm would you use ( or any broker use ) to transform it, so you get a decent and meaningful dataset? It's not so easy as it seems.
Reducing the amount of accuracy of prices seems like an odd thing to do, what exactly is the goal here? Generally speaking such a transformation is inherently lossy (it destroys information) so there are going to be multiple ways to do it and none of them will be perfect. You need to figure out what is meant by "decent and meaningful" and then try to find the best way that satisfies what you are trying to achieve. If the issue is just to have fewer numbers displayed on the screen in order to reduce clutter, you could simply round (to the nearest) all the prices to however many digits you want. To keep limit orders behaving like you'd expect, bids need to be rounded down and asks rounded up. If you want to have fewer ticks displayed you can do various things to merge same-price same-time trades, or even quantize the time dimension to get more reduction.
Unfortunately, the reason is neither of what you call but I appreciate a lot your help with interesting hints. I'll keep exploring other methods. Once again, thanks. Regards
For stock data there is the CTA and UTP feeds which consolidate raw tick data into a standardized stream. This may be what you're talking about?
Hi Polygon, Unfortunately, it is not. But at this moment I've designed some algorithm to consolidate according to my goal, still not perfect. Thanks