I'm also working on my own trading system, and I want to tackle this problem using the NASA way - have multiple pieces check each other. So I will have a second independent little process which will also connect to the same trading account and monitor simple things, like ensuring the total net position does not go over a certain limit, or that there are no more than N orders in 10 minutes for example, or more than M positions open simultaneously. You can also check that the actual open positions correspond to what the strategy process thinks it actually holds. As a bonus you can run this checker process on a different machine/network provider. Besides the checks in the main strategy, this will ensure that whatever weird bug occurs, nothing really bad can happen. You can also download RightEdge and look inside the SDK help file. It has lots of classes for handling stuff like Position, BrokerInterface or Strategy. I looked at these interfaces to get a feel of how you go around designing such a piece of code. EDIT: My approach would have avoided this situation, when an algo went insane: http://arstechnica.com/business/new...se-click-choked-the-nyse-cost-a-bank-150k.ars
After several iterations with these types of issues, my final design is that the strategy code makes no assumptions about what execution you will get. For example to buy at market, instead of sending a market order and assume I have a position, I send a limit buy above the ask to the execution module. When and if I get a fill, I queue up a "filled" message back to the strategy. The strategy may receive a "filled" message for an order it already canceled. As a consequence, the structure of the strategy is more complicated, but more robust .... at every tick it must evaluate what is the current position, what is the desired position, and place the appropriate orders. The point is that having a rejected order may be rare (and usually avoidable) but a partial fill happens all the time. I would rather have the strategy logic handle a partial fill than have the execution module blindly grab the remaining shares at market. The execution module design is to throw orders onto a queue and then do the best it can. It handles issues like if an order seems to get hung up in the system. It groups the orders from an originating strategy, and makes sure all cancels are confirmed before sending out any new orders. This ensures that if the strategy cancels a limit order and moves it a penny lower, you won't ever get filled on both. The contract with the strategy is that the orders will be submitted synchronously in a FIFO manner. This also solves margin headaches. If your strategy keeps within margin, the execution module will keep within also. The queue has other advantages. In the case of HFT, the strategy may submit a cancellation for an order that is still pending on the queue. I can just take it off the queue and never touch the market.
best practices for order handling basically start and end with one rule, never assume anything... unless of course, you think you can get away with it. each order req/mod should have 2 states, pending/confirmed with handlers if the pending should fail/reject. monitoring of 'pending' duration for possible failure/stall is a good idea. sending test orders sent every x interval for exchange failure is a good idea. don't rely on heartbeats. backup state mgmt/broker in case of line failure to exch, but data still functional. ie, line goes down, limits are out, data indicates x exposure... get your hedges on. exposure and risk are biggies. especially if your broker module is weak here, as you mentioned. don't skimp. there should be failsafes everywhere. there's nothing like unleashing beyond ridiculous positions into the market because of faulty data or exch error etc. to make you reconsider the benefits of redundant checks. there's a ton of things you'll need to handle. you won't think of them all. i've been at it awhile and i find/come up with new ones all the time. the most important things you learn are usually the most expensive, but as long as you're making money and lock down the risk aspect of the order module, you should be ok and can incrementally handle issues as they arise. good luck
i just keep track of open orders (and cover orders) and keep track of positions(an some calculations) from fills coming back. when i place an order i do some checks and when I get back a fill I do some checks.
Yep, this is essential. I have a set of monitoring algos running in a separate process which execute an emergency shutdown of the strategy should it breach any of the operational rules. Even more essential in my mind is that any algos are monitored by a human at all times they are in operation. By definition it's the scenarios you haven't planned for that will kill you and you need a set human of eyes (that fully understand how the algos should operate) to always be watching. The speed of silicon processing will always beat the speed of carbon processing, but carbon still wins when it comes to interpretative decision making. P.S. Love the article BTW. Bet the kid who slapped that 'enhancement' on the UI was ripped a mighty big new one. Not that surprising though as it closely mirrors some of the stories I've heard from my mates who work The City. Some of the development practices adopted by even the biggest institutions are truly shocking. Talk about systemic risk...
Agreed. This is essential. Until you receive a confirmation report for a particular action you must assume it has not been carried out. E.g. Any open order for which a cancellation has been generated is still working until the cancellation confirmation/rejection is received. This one of those scenarios where a rejection can be legimately generated in the normal operation of a strategy. As long as the messages from the exchange are returned in the correct sequence and the fill report arrives before the cancellation rejection report then the reason for the cancellation rejection will be obvious. If they come out of sequence then the handling code will get a bit more complex. Anyone know what the chances are of these things coming out of sequence? In a really fast moving market practically anything can happen. For example, you could place a marketable buy limit order 5 ticks above the best ask and still have it rejected for breaking the CME's price banding rules if there are transactions printing all over the place. In these sorts of scenarios I just put my strategy into emergency shutdown. Take whatever hit I have to in order to get flat. I want to be completely out of the market at times such as this.
Totally agree. The potential for screw ups of cluster fuck magnitude are enormous when operating in this field. This is what gives me the fear as I lay awake in bed at night. Seriously. As a result I have massively simplified the orders I am using in my strategy. It may hurt my profits slightly at the end of the day but I am far more concerned about managing risk that squeezing a few more % out of the strategy. With regards to order placement/modification/cancellation rejection messages I operate a whitelist policy. If I have recognised a particular rejection scenario as being valid then I handle it. If it is not handled then an exception is thrown and as a result the strategy initiates an emergency shutdown (cancel all orders, wait for all orders to reached closed state, execute market orders to flatten each position, alert human operator to perform validation that strategy is flat). I've focused in this thread on rejection messages as they are the simplest manifestation of these concurrency problems. Obviously there are plenty of other scenarios not involving rejections... Does anyone reckon it might be worth sharing a few of the more generic scenarios that can crop up operationally between us here? Could alert a few people to weaknesses that they aren't handling in their own strategies. Or do you reckon it's more a case of these scenarios depending too heavily on individual implementations? Thanks. Am going to need it. And thanks to everyone that's replied. I don't think there are that many of us here on ET working at this level so it's good get some productive dialogue going. The diarrhetic quality of most of the content on this site leaves me in despair much of the time.
I'd list to see some ideas of others on exception type situations. I'm sure there are some I haven't thought of and others may have. We should be able to learn from the exceptions identified by others, even if they are situation specific.
Why not post a UML or interface/abstract classes of your proposed designs? In my experience, there are trade-offs and no one solution fits all. For example, if I need the system to be as fast as possible, I try to minize the object oriented designs that slow down orders, cancel and replaces, and code straight to the API I am given. The downside of course is that if my route changes, I have to rewrite this piece of code as many times as this happens and anything that directly assumes this design. I have made the decision to live with this in this extreme case of need-for-speed/latency. If speed is not critical, then I attack the problem with all the design patterns conceivable to make the code as decoupled as possible. The one thing that I try to do even in this case though, and it costs me a litle bit, is to push state as far as possible into the API. So what I don't do is keep my order state, and the API has it's own idea of order state. I have found that not only violates general principles, it is a great way to get what your position/order state is and what you think it is, out of sync. I still think you should post interface/classes in say C# so that design/code can be (re)viewed.
I don't depend on a cancel rejection message. I log the message and, unless the order is confirmed as 100% filled, it tries to cancel again. I have received a cancel rejection both before and after a fill. I have received a cancel rejection with a blank reason code. My code immediately resubmits, and it cancels no problem. I do depend on "order confirmed (# shares)", "order canceled (# shares)", and "order filled (# shares)". Using the Genesis GTP protocol these messages have been reliable. For a given order ticket ID, I assume the order is in play until (order confirmed # shares) = (order canceled # shares) + (order filled # shares). Once the order is out of play, I can send the next orders (FIFO queue).