Hi, I am trying to reconstruct the limit order book from the NASDAQ ITCH feed, and am confused on how to treat the hidden order execution messages (The "P" Trade message for non-displayable orders). Do these orders impact the limit order book? How? If they do, how would one go about updating the order book from them - given that these messages no longer provide the order ID or the direction (buy/sell) of the hidden trade? Thanks a lot !
@basic_trader Hidden orders are just that, hidden. They're designed to interact with matching orders in NASDAQ's order book, but not reveal its presence, size or price to the public. Any match that happens, gets published as a sale to consolidated feed, but original hidden order never gets published outside closed NASDAQ order book. The only way you can treat these messages would be by only showing the prints in time&sales. The state of the order book does not get affected by these since there is no corresponding order add/order modify event with reference number. NASDAQ is the only party that knows of hidden order's existence, but they're not publishing any information to public via ITCH.
Thanks for the explanation. Can a hidden order (e.g. sell) be matched against a non-hidden order (e.g. buy), or only against another hidden order? If it is matched against a non-hidden order, then I would see an execution message for that non-hidden order.
@basic_trader Yes, regular non-hidden order can be executed against a hidden order (most are done this way). Consider this: Buyer A enters a hidden order for 25,000 @ 50.25. This order isn't reflected in public order book. Seller B enters a regular sell order for 500 @50.25. NASDAQ sends Add Order for 500 @ 50.25 SELL with ref number 12345. Book gets updated with new order. NASDAQ matches 500 shares and sends Execution message for 500 @ 50.25 with ref number 12345. Book gets updated with removed order 12345. Buyer A is left with 24,500 shares at the bid, but is still invisible to everyone.
So I understand that when reconstructing the order book from historical data, I have no choice but to ignore the hidden execution messages. As they do not change the state of the order book. I guess what troubles me is that I feel I am missing something in my back testing when ignoring these messages.
@basic_trader Yes, you really don't have any other use for these messages now. It used to be some time ago that these trades contained a valid order ref number, as well as buy/sell indicator. Anytime a match happened, NASDAQ would send hidden trade event with same order ref number and indicator to ITCH subscribers. If you monitored for these hidden trades, you could pick up that there was a hidden large buyer or seller, and front-run these orders. When NASDAQ posted these trades to consolidated trade tape, this identifying information got stripped out. You basically gained an inside knowledge if you had ITCH subscription, that there was a large player, vs if you just had consolidated tape. NASDAQ was forced to zero-out order ref numbers some time ago because of this unfairness. They also removed buy/sell indicator recently because that could've been gamed as well by HFTs. For fully transparent markets, hidden orders should not exist. But that's wishful thinking..