I'm not using hiddens a great deal right now, but since last August I've worked tens of thousands of them and gotten filled on at least several thousand ... my platform notifies me immediately when the NBBO gets close enough to my personal BBO (hidden or not), and it squawks extra-loud if I'm crossed or locked, so I would see what you describe 100% of the time ... I don't recall even one time when your experience has happened to me, except MAYBE the first day or two (see next comment below) ... in other words, SMART routing has worked fine for me with hiddens, BUT: The first day or two I tried hiddens I was using standard SMART routing and it was horrible - orders that should have been on NYSE kept getting stuck on ISLAND ... this is grossly unacceptable to me, PLUS there is no way to know when it happens without manually viewing 1000+ TWS market data lines every 5 minutes, since IB in their infinite wisdom has chosen to not let us have API access to the exchange on which a SMART routed order is currently being worked. Long story short, in order to experiment with hiddens I had to choose the global SMART routing option to always send non-marketable limit orders to the listing exchange first, before applying any other SMART algorithmics. Contrary to what you might guess, this worked great. Based on my thousands of observations, I believe the only downside is that I miss a few decent dark pool executions per day - only a blip on my p&l screen. However, I would understand completely if many traders found this SMART setting to be odious and unacceptable for their own trading...
Followup information for anyone who read my post on page 14 regarding hiddens: I have completed my experiments using a non-IB DMA account, sending NYSE-directed hidden orders into the back (i.e., at least a few levels back from BBO) of the NYSE order book ... IB is not misbehaving, it is really true that some predators know all about all my hiddens, at least whenever they care which is very frequently. So the bottom line for hiddens is: No matter where they are directed, any predator who cares knows all about them if they ever become NBBO ... and for NYSE hiddens, any predator who cares knows all about them no matter where they are in the order book. What a wonderful world we live and trade in!
Just to clarify what settings you're using...I see 5 SMART options in my TWS configuration: -Multipurpose -Highest Rebate -Primary Exchange -Highest Volume Exchange with Rebate -High Volume Exchange With Lowest Fee Currently, I have Multipurpose checked as well as Seek Price Improvement. If I'm understanding you correctly, you use Primary Exchange and have Seek Price Improvement unchecked, is that correct? It's still not clear to me why any of these settings would allow my order to sit there for 5 minutes with the NBBO ask price .04 less than my bid, but who knows. I've sent the order details to IBSoft and they're looking into it - will be sure to report back any relevant info.
I had "Primary Exchange" checked and nothing else since I started messing with hiddens ... I think with my extremely high level of knowledge and genius I can answer that question: Because IB's code is full of nasty, smelly, disgusting bugs that they don't care about fixing. Please let me know if you desire more profound revelations. Joking aside, I would appreciate reports on your progress with this issue: I want to change my settings back to "straight SMART" on a permanent basis ASAP ...
Heard back from IBSoft regarding marketable hidden orders not routing out to execute. It appears there's a bug and the orders don't work correctly (route out) when "Seek Price Improvement" is checked. They are working on updating the code. I will post back here again when it's done.
Just got a report back from IBSoft that the issue with Hidden orders not routing has been resolved. I haven't had a chance to test it myself but if anyone else still has issues, mention it here and I'll make sure he knows about it.
Guys, what about the futures exchanges, especially CME ? Do this kind of abuse happen there also ? In my limited experience, the abuses seem to be lower there. There might still be broker related abuses like internalization or routing your order to 3rd parties.
I have seen people write bucketshop in relation to futures but I did not think was possible with futures. There is one legal place to trade globex contracts (well some still have a floor portion also) right ?
Yes, I think this is true by a 1974 law: http://en.wikipedia.org/wiki/Commod..._Act_of_2000#The_exchange_trading_requirement In practice I think this greatly reduces the problems faced by equities trading (internalization, payment-for-order-flow exclusive deals with wholesalers, fragmentation). One negative aspect of the futures case is that the exchange is a monopoly. Also, although there's far less "bad behaviour" on futures exchanges, there appears to be some, and the exchanges may turn a blind eye towards it because it's often their largest revenue-generating customers who are doing it. But overall, I think the costs assoicated with the futures markets are offset by the advantage of having a single book everyone goes through, which is huge. The biggest issue underlying a lot of the problems with equities is that retail brokers like to shave a little off of their clients with every trade, and this is currently legal, albeit not directly. If they are so inclined (which, it seems, many are), this is easy for them to do through "smart" order routing systems coupled with internalization and/or payment for order flow deals with wholesalers, unless you as a customer are using DMA (directing your order to an actual exchange so that it doesn't go to a dark venue, the broker/dealer's own pool, etc.).