I had a bug hit me yesterday that happened in a kind of freak scenario I didn't envision, despite having programmed in numerous safety checks. I'm just curious whether any of you have some horrible automated trading disaster stories. Knight probably takes the cake in this department (so far), but the goal of the thread is to maybe highlight obvious software checks we should make but somehow oversee or don't implement properly.
Sure, I'll tell you my bug. The strategy of mine posts multiple passive orders to get out. Usually, when one gets hit, I cancel the others and move along. Occasionally, someone sweeps the market, and I'll get overfilled. Normally, when I go into an overfill situation, the strategy turns off quoting, switching into a remove liquidity mode, and just waits for the next best time to remove shares. Then it stays in this situation until I look at the situation and flip a switch back on. What happened was the broker server delayed fill messages for an extremely long period of time because the market was moving super fast, so the passive quotation engine was quoting in a fast market while the signal to flip into "escape mode" didn't arrive soon enough. During most markets, even somewhat fast markets, the fill notifications happen within (at most) a few milliseconds. For whatever reason, yesterday, they started taking seconds. I was receiving fill notifications while the NBBO had moved a full 8-10 cents away from the fill-price. When I got into this situation, the only protection I had left was "max outstanding orders", which I had set to around 20. The problem was, even with 20 outstanding orders, that was still a lot of shares, and on those outstanding orders, even those fills were delayed. The bug-fix is just to enforce an even harder cap on "potential exposure" on the code that does the passive quotation. It seems so obvious now, but I didn't have it in there because I never saw this situation come up before.
I'm imagining I'll get a lot of flames on this thread in advace, so I'll admit, my bug is stupid. I just think I underestimated how bad order server and fill notification lag could get. This is more a class of bug that goes into the timing / race-condition class of bugs. I usually fix these things as they arise, for better or worse. My guess is people with superior technology or who don't have third party dependencies have better hard guarantees on how long it takes to report a fill. Or maybe they don't. Either way, I consider myself lucky for having gotten off cheaply with such a stupid mistake.
Just an obvious one for low level programmers like me. When you use excel speadsheets with IB, just make sure everytime you open it there aren't any active order placement DDE links in it ( especially when your computer froze and you are restarting it ). Nothing worse than placing 250 limit orders based on yesterday's prices...LOL
Yeah, agreed, but I can't exactly go looking into the plumbing of why my broker's order server took 3 seconds to report something. At best, on my end, I can program checks to not make a bad situation worse. I still consider it a bug, though. My programmer ego was definitely bruised yesterday when, despite failsafes, I ended up acquiring more inventory than I thought was possible with my code. I can only imagine how much worse it would have been if I had been up getting coffee or taking a piss or something stupid and had been briefly away from my terminal. I think I'm going to keep a bucket next to my chair from now on.
I used a third party interface between a charting package and the broker. I discovered that if price was under ten cents it added three zeroes to the price! I never got to production with that bug in place but it could have been a fat finger error that I never would have forgotten.