Best practices?

Not necessarily. Not all APIs I have used keep local state for me. For example, many times I am forced to use FIX. What I am saying is that even if I kept local state and I crashed, and the API offered me a chance to sync my positions/orders with the broker/clearing firm, I would always choose that over local persisted state. So the case of using local persisted state in my experience is of limited value, if any value at all.

Where I find value in persisting local state is to prove order precedence, or trading through my price, etc, when there is a dispute. But there are tools out there that already do this, e.g. Bloomberg...

 

My specific algorithm is to maintain a "float stupid" per symbol. If stupid crosses above some threshold, there is a problem. So to buy AAPL at limit X and then sell AAPL at limit (X-0.2%), add 0.2 to stupid. Buy AAPL at limit X and sell AAPL at limit (X-0.4%), add 0.4 to stupid. Buy AAPL at limit X and sell AAPL at limit (X+0.2%), is a nice trade so add nothing to stupid. Stupid is decremented over time, i.e. a stupid sequence is OK over 10 minutes time, but not OK over 10 seconds.

 

that's a very big 'and' which you're relying on. in the case of line failure you won't get that chance. your only hope is your redundant internal state and your redundant execution venue. or are you going to call the exchange to figure out where you're at so you can get your hedges on? ridiculous.

here's another one for you. a couple of years ago the data center we were hosted at had a fire. first thing the fire department did when they got the call was to cut off the power in the building. of course, the backup generator failed immediately and the ups's were damaged in the fire. what saved us? our internal order state which is remotely monitored/recorded at colo#2 at a different data center. hedges were on to secondary execution venue as soon as server1 lost connection. servers never did come back online that day. we had about 30M+ in positions that afternoon and prevented a 6-figure loss... but i guess persisting order state is of limited value.

it would do you and your firm much good to come to the realization that risk management is also an IT problem.

 

I don't follow the situation you are describing. If what line goes down? The one from my location to the colo machines? The ones from the colo machines to the broker/exchange? I am assuming from the colo to the broker/exchange.

How does saved order state help you. There could have been dozens of fills from the time the line went down. If you hedge based on what you have in your saved state, it could be a completely different position than what you think you have? If for whatever reason your system is not coming back up, the only way to make sure your hedges are accurate is to call your broker and or exchange and first cancel all open orders, and then work your way backwards from estimated time of failure to now to figure out that your saved state didn't miss any new fills in between when you went down and your saved state. Otherwise you are just guessing that no fills occurred between when the line went down and when you saved state. The ideal situation is to be able to sync from somewhere what your current position is automatically, not to rely on saved state.

This is only one solution for fail-over, one whose design I do not advocate. Another solution would be to have identical systems running at two separate colo locations. It could be an active spare or probably way easier is a passive spare. On fail-over, the first thing the fail-over system does is sync positions with the broker/fcm. This is far more robust than your internal order state saving mechanism for reasons I have stated. If your broker/API does not allow for syncing of positions, then yeah, you have to keep state yourself and hope for the best.

 

A few things occur to me...

1. If you rely on a stateful API the problem of maintaining state is just pushed one rung further up the ladder to the API technology provider. If they lose their connection to the exchange then they are going to slip out of synch just as you would if you were maintaining internal state.

2. In the event of a connection failure between you and the API technology provider you have absolutely no idea what's going on. All those API requests to replay order state are going to fail. The advantage of maintaining local state is that at least you have a snapshot of that last known configuration. Alright, the config may change significantly whilst the connection is down but without it you are totally in the dark.

3. I just don't see how you can build a system that doesn't maintain internal state if you are using a stateless messaging API (e.g. FIX). At the lowest level the exchanges maintain the state of orders in their engines and then spit out data messages regarding any state changes. In order for an app to have any idea of the current state of play, at some point these messages have to processed and the information contained within them persisted in some manner. Either the API does this and exposes the data via some sort of replay functionality or you do it. Either way, by definition this is maintaining state. Otherwise, how can you have an application which cannot query the state of the orders it has generated? State must be maintained somewhere in a format the app can query as required.

Maybe I'm missing something obvious here, if so put me on the right tracks...

 

no, you're not. nitro's a broken record. first its persisting locally is bad, then when questioned specifies only if the api is saving locally, then redacts and states it has little value even in stateless protocol's like fix, then apparently concede's in his last post that it might be a good idea but only if the api doesn't sync. whatever.

nitro, how are you going to sync box2 if your broker's down and you have no local state there? you can't sync, that's the moral of the story. your solution of reproducing the same stateless system as your 'backup' on box2 in case of broker failure will solve absolutely nothing. if you don't have local state on box2, you're not trading period. your solution is to call your broker/exchange to get state? give me a break. i suppose if you're managing 10 positions you could get away with this. just write them down on a sticky note, right? try making markets on thousands of stocks/options and try 'working your way backwards' with this 'fail-over design'.

redundant broker/api fail-overs, redundant exchange fail-overs (cme down, hedge the etf's), redundant data feeds, redundant local order states, redundant hardware. if you're managing significant sums, and you're not doing this, you WILL get bit. it's just a question of when.

 

First, when talking about this stuff, it would be helpful if you differentiated between order state, position state, and marketdata (subscriptions etc) state.

Look, you have a class like this.

Code:

public class MarketData
{
    ThirdPartyAPI thirdPartyAPI;
    ....

    SubscriptionEntity SubscribeSymbol(Security s)
    {
         return thirdPartyAPI.SubscribeSymbol(...);
    }
}

as opposed to

Code:

public class MarketData
{
    ThirdPartyAPI thirdPartyAPI;
    Dictionary < Security, SubscriptionEntity > subscriptionEntities;
    ....

    SubscriptionEntity SubscribeSymbol(Security s)
    {
         SubscriptionEntity se;
         if(false == subscriptionEntities.TryGetValue(s, out se))
             return thirdPartyAPI.SubscribeSymbol(...);
    }
}

The point I am trying to convey is that you are keeping state in two places in the second case where the first case simply delegates state to the API (if it keeps it). This is poor design with no added benefit (this is a simplified case. Maybe you want to have multiple users. Then you have to keep a count of references to the subscription, so that you only unsubscribe from the thirdPary if there truly are no subscriptions. It is just a simplified example.)

In the case of OrderState, if the API is doing it for you already, same comment. It doesn't mean that you shouldn't duplicate some code in certain cases, it just means trying to delegate the state as much as possible to the owner of that state, to have lazy evaluation instead of eager evaluation as much as possible. The more the system needs to be fast, the more duplication that needs to occur. This is nothing more than saying you should cache state locally (although I have seen cases in functional programs where there is no state be faster than C++ threaded implementations that keep state. There is no one size fits all). But notice how perspective changes when you make explicit what you are doing. Now everything you learn about the perils of a cache immediately come to mind.

Your not understanding what I am saying. You are assuming that the API itself has to sync with the remote server. If the connection between the API and the provider goes down doesn't mean the API doesn't have the current state. Obviously someone is keeping state. either locally or remotely. I am saying if the API already keeps state locally, why would you duplicate that and offer another way that state can get out of sync? If the API has to query a remote server for that information, then of course you have to keep local state. I already said that. What I am saying is avoid duplication of local state.

Right. In the case where there is no local state maintained like FIX, which I mention above in another post, is another case where you have to do the local housekeeping yourself, you need to maintain local state yourself.

Finally I am also saying that in exceptional cases like crashes, the last thing I want to do is rely on local state if at all possible. What I do is on login to the broker, I request by some mechanism the opening position coming into the day (that often comes in the morning in some manner that you have to parse and store in SQLServer or whatever), and then request all intra-day orders up to now. I then rebuild my position and act only on that. Some APIs will send you orders you missed so you only have to update those on your local state. They have a software order-ACK. But even in this case I might still prefer to relive the days orders and fills. It takes 1 minute instead of 1 second. The point is I ultimately trust what my broker says my position and my order state is in exceptional cases. If given the choice, I always rebuild state from a broker on major exceptions.

 

BTW, notice there is another reason that you have to keep position state across day boundaries: you need to reconcile what you think your position is with the brokers. Often times, you get positions that don't belong to you, or the statement doesn't have a position which you know you have.

There is a strong argument for keeping local state all the time no matter what, since there is no real reason errors can't happen intra-day either, although I have yet to see that happen.

Another reason you want local state is one that doesn't always occur to people. I was at a prop firm as a developer. A trader complained about a fill or an order and he didn't know how it got there. It was a big loser and the head trader was pissed because it fell outside of what he was supposed to be doing (hedging SPX options with ES). I was called over, and showed all the states the program had gone through, including a mouse click from the front end to execute a fill. It never occurred to the trader that we were keeping this information. Whoooops!

Finally, the more symbols you are quoting, and the more automated it is, since there is no human there that probably can say, no way we have that position on (since the computer can be sending 20,000 quotes a second on 100 symbols), you have to keep local state on everything. Sad but true.

 

How do you deal with going long/short at the same price?

Do the pros have a short account seperate from a long account?

Let's say system A has a limit buy 1 ES at 1105 and system B has a stop limit sell 1 ES at 1105.

Based on execution both systems need to place stops targets etc. How do you deal with this?

 

Here is something from my own designs that I have found lyes at the foundation of writing general systems.

If you look at the class above, MarketData, it has a method

Code:

SubscriptionEntity SubscribeSymbol(Security s)

Hidden in this is quite a bit of design. For example, the thirdParty API may not understand what a Security is, and the thirdParty API may not understand what a SubscriptionEntity is. We need a transparent, plugable translation framework that knows how to translate to/from our classes and the third party data structures.

So what I do is I use something like the following that sits between the thirdPartyAPI and my own interfaces.

Code:

namespace TranslationFramework
{
    public interface ITranslationProcess< To, From >
    {
        To Convert(From from);
        From Convert(To to);
    }
}

When I am implementing a new thirdPartyAPI, all I have to do (there is quite a bit more design, but it rolls off your fingers once this basic idea is in place) is implement these translators, and using Dependency Injection, "plug/inject" this into the flow "down" from my code into the thirdPartyAPI when I am passing parameters, and "up" from thirdPartyAPI to my code when returning values. All the translation is then transparent and completely dynamic.

Since it is all wired using DI, unit testing with mocks is trivial, and decoupling is maximized.