The following case is a bug described by IBj (Interactive Brokers) which caused a client to lose money. IBj (in response to a bug which sent an unexpected order out of the blue): I have attached an annotated order trail with ID data overwritten. The audit trail has real world info such as time points for various news events as well as the market price for GBP at various interesting times. For those interested enough to follow this, there is a chart attached in MET to match the time descriptors in the audit trail. Here is what happened: 10:30:01 Client created a basket of GBP currency related orders in spot and futures contracts. Total value across 19 orders was roughly 31M GBP, all selling GBP. This was submitted at 10:30.01 and may have been tied to trading UK economic news. The client clicked submit for the basket twice in very rapid succession creating the following timing issue: The orders from basket set 1 were being credit managed and routed when the "second" basket comes in. The orders from the first submission are in transit through our system and out to the various liquidity providers and the downstream processes (closer to the exchange) had not acknowledged their state so when the second submission hit, the upstream order control process believed it still "owned" the orders and interpreted set 2 as a modification. When this happens, the order IDs are incremented and the old order IDs become stale/expired. Now it gets messy. We start to get the acks from the downstream processes confirming they have received the orders from set1 and are forwarding them on down the pipe. But as the order control process thinks these orders are stale and have been replaced with modifications, we are now managing a set of orders with undetermined state. As one sees in the attached audit trail in brown, there are a series of messages stating "NoSuchOrder" which is what we record when a downstream message comes back with an unexpected order ID we don't know how to handle. The client's GBP futures order was one of these (see 10:30:02.877). We are in a messy state with 6 of the submitted orders. The others escaped the timing snafu. I did not bother to trace the ultimate resolution of the other 5 since they either never executed, the order controller resolved the indeterminate state, or the client accepted them. But one of the GBP orders was left stuck in an existentialist desert, neither active nor cancelled. Times passes ... 17:46:46 for unknown reasons there is a TWS disconnect. Could have been an IB disconnect, an internet hiccup, or something on the client's side. It actually doesnt matter. This is just the next banana peel. 17:48:57 TWS automatically reconnects and the order control process gets the computer equivalent of a wake up nudge and all of a sudden rediscovers the stranded GBP futures order. As it is marketable, it is submitted and gets filled in a series of small executions at roughly 1.9929 18:08 to 19:12 various stories out of the UK runs sterling up a bit to ~1.9945. 19:38 there is another disconnect/reconnect but based on the speed, it is presumed that this was an automatic reconnect 19:55:57 Client submits multiple orders in various currencies. This is the first time we can know with certainty that the client is using and attending the TWS. Had he covered the GBP position at this time, the loss would have been roughly 12000 USD. 20:07:31 through 20:10 Client submits multiple orders for GBP futures on the buy (covering side) although the bids seem to be far away from the market. Had the position been covered here instead of just being bid, the loss would have been ~8700 USD. 20:19 As widely expected, and on the announced schedule, the Fed cuts interest rates; the 50 bps is the surprise and the futures rocket up from 1.9954 to 2.0050 in seconds, retreat back to 2.0000 and then proceed rapidly again up past 2.0080. 20:26:19.36 Client commences covering in earnest with initial fills at 2.0021. 20:35:51 Client finishes closing the 66 lot position with the futures trading at 2.0090 SUMMARY / ANALYSIS 1. IB's client agreement says that the client is responsible for any order he/she submits and, strictly speaking, the client did submit these orders. But sorry to disappoint all the people with their hands on the speed dial to their attorneys, IB really tries to run its business based on business principles, not legal nuances. We live in a litigious world, so protective agreements are the universal way to define exposure limits around which we are always free to improve. 2. The double click of the client was an error but so what. It is not reasonable to expect someone to recognize this extraordinarily arcane situation. The TWS was patched to prevent this about a week after the problem was reported to the developers. 3. IB will indeed compensate the client for loss. We will need to decide how to interpret the activity from 19:55 onwards to determine the amount. It is a problem that the risk was not covered when first recognized. IB's standard position in all trading issues is "manage your risk" because the market will not sit around waiting while we research a problem. 4. We absolutely dropped the ball in responding to the client. The problem occurred on 9/18. He contacted us a few hours later and then contacted one of our developers with whom he has previously communicated on October 5. It was escalated and it actually even got to me (as I just am discovering going through the contact history now) but I seem to have misunderstood the problem and passed it to the wrong person. It simply got buried as new issues piled in behind. Not good. I apologize to Zzzap that we did not resolve this more promptly. Audit trail: http://www.elitetrader.com/vb/attachment.php?s=&postid=1663721
I would like to raise some questions for discussions regarding this issue. Please be rational and objective. No flame war please! 1. The client clicked on the "transmit" button twice to send the marketable order [*]. However the order has been reduplicated by mistake. As far as I know, one order is sent only no matter how many times you click on the "transmit" button. Furthermore the order is stuck at the backend so you can't view that order on the frontend of TWS. I would like to know more why double click on "transmit" button can lead to the creation of a duplicated order and that order being hidden and stuck in the TWS backend. *: Although the OP didn't specify, the basket contains a marketable order. It will be executed once acknowledged. Apparently it got stuck in the backend so nothing happened. 2. TWS disconnected and reconnected several hours later. The stuck basket order was revived unexpectedly and executed because it was marketable. I think the problem can be avoided if you dont use market orders. I always don't send market orders to avoid any potential problems occurred. I will only send marketable limit order when I want to chase the market. However TWS feels like using market orders. If you want to attach the stop to an existing order, there are two options namely Attached Stop and Attached Trailing Stop. Both stops only attach market stops. It is weird the option of attaching limit stop doesn't exist. 3. I'm afraid there are some wrong assumptions regarding this issue. IBj assumed he recognized an unexpected position was opened since the client sent various orders via the TWS platform. However the presence of the client didn't mean he must recognise the position. If he doesn't expect a position to be exist, how come he will keep checking it (just in case a position will pop up all of a sudden)? Does IB expect him to check positions every once in a while to see if there is some bug which opens up some positions? Furthermore, did the revived order show up again in the market data row of the TWS main window? Did the open position show up in the market data row of the main window? Did the open position show up in account window or execution window? If the order or the execution is not visible in the TWS frontend for some reasons (eg bugs), it makes it very difficult to realise such a position exists. 4. Unfortunately the client still hasn't received any compensation for about 2 months. I hope IB can resolve the issue and figure out how much as soon as possible. However, as I said, the client may not realise the existence of the position, especially when there is a bug to hide its existence. Full compensation should be made in this regard. The experience is really too painful for the client. 5. It is good to see IB has the bravery to admit its mistake (and the bug). It would be much better if the complainant doesn't need to bring this to the public before IB takes the initiative to solve the problems. 6. For clients who are still using IB/TWS, what measures do you take to make sure there is no unexpected order being executed or position being opened? TWS itself may not be reliable because there may be bugs which hide the order execution (like the reduplicated order being stuck in the backend) as a matter of fact. Thanks for the discussion.