http://www.lrb.co.uk/v36/n17/donald-mackenzie/be-grateful-for-drizzle -----------------------------" By the time the old code was found and switched off, the firm [Knight] was on the brink of bankruptcy. Such events don’t always become public. In a New York coffeehouse, a former high-frequency trader told me matter of factly that one of his colleagues had once made the simplest of slip-ups in a program: what mathematicians call a ‘sign error’, interchanging a plus and a minus. When the program started to run it behaved rather like the Knight program, building bigger and bigger trading positions, in this case at an exponential rate: doubling them, then redoubling them, and so on. ‘It took him 52 seconds to realise what was happening, something was terribly wrong, and he pressed the red button,’ stopping the program. ‘By then we had lost $3 million.’ The trader’s manager calculated ‘that in another twenty seconds at the rate of the geometric progression,’ the trading firm would have been bankrupt, ‘and in another fifty or so seconds, our clearing broker’ – a major Wall Street investment bank – ‘would have been bankrupt, because of course if we’re bankrupt our clearing broker is responsible for our debts … it wouldn’t have been too many seconds after that the whole market would have gone.’ What is most telling about that story is that not long previously it couldn’t have happened. High-frequency firms are sharply aware of the risks of bugs in programs, and at one time my informant’s firm used an automated check that would have stopped the errant program well before its human user spotted that anything was wrong. However, the firm had been losing out in the speed race, so had launched what my informant called ‘a war on latency’, trying to remove all detectable sources of delay. Unfortunately, the risk check had been one of those sources. "------------------------------- When a single "-" can threaten a large investment bank or even the whole market, something is seriously wrong. As I have said before, this speed chase rewards the most reckless HFTs (those that leave out error checks to save computing time), until they spectacularly fail and threaten the system. Brokers and clearing firms need to check margin for ALL customers big or small before they accept any order. Exchanges need to check clearing firms' margin as well before they accept any trade.
Lots of people here have probably worked in software development. I recall a testing and debugging effort involving maybe 50 or 100 people. It was a complex device for doctors to use to make adjustments and collect data from implantable medical devices. The company had a pretty keen sense that the quality had to be there, we understood that people's lives were potentially at stake. We collected 700 bugs a week for many, many weeks. We uncovered all sorts of problems including one engineer that simply faked test data to show that his software worked. I write some software. You cannot have much ego left after a computer shows you dozens of times per day that you get things wrong. I make so many mistakes that it leads me to question my very ability to exist at times. Fortunately it's just for trading on my small account, almost a hobby really, but I take a lot of pride in getting things right. I make sure that I have a visual on everything the code is doing, on the charts and the log. I can first look at what the software is doing, just give it visual inspection by having it draw on the charts, then I can run it on an IB play money account and see how it interacts with the brokerage and to some extent, a market in realtime, then I can move to trading the least amount possible on a real market and allow enough time at that to get some confidence that nothing odd is going to crop up. Essentially, anybody that went live with a minus sign in place of a plus sign... wow, they need to be watched a lot closer because they are devoid of some necessary concepts.
I bet that human errors in manual trade entry and rogue traders cost even more than the software bugs. The $3 million loss is nothing compared to the $1 billion loss by Nick Leeson at Barings Bank, or the $2 billion loss by Bruno Iksil at JPMorgan, or the $7.2 billion loss by Jérôme Kerviel at Société Générale. The list of these non-HFT incidents is long.