I've already posted here some rant about data providers from the black box developer point of view. Today I have some free time, so this time I will complain about brokers. The black box trading differ from human trading by one very important aspect: error handling. It means, that in case of ANY problem program should either know how to solve it, or if it can't, then get to human and ask him for fix The approach I took with my black box was this: I don't want to babysit it. Basically, I want this program to be automated as much as possible - to the point of automated money farming (or printing ) machine. I mean, I should be able to leave it running for months without me even checking "how are you doing?". I know that this will never happen, but it is a goal. So, the best way here would be to not react to errors, but prevent them from happening. I've already coded for all sort of errors that I can think of - connection failure, harware failure, etc. Now we are at a point of prevening broker errors. And here comes unpleasant discovery: there are NO brokers properly suited for black box trading. Well, at least in my price range. First of all, let's start with an API. The sort-of-standard right now is FIX gateway. From reliability point of view, this is probably the best API at this moment. The reason is - each message has a number, and both sides are very careful to process them in order; and if any number is missing, they will re-request for it as long as needed. If it never received, and other party does not provide an excuse, then no further input is taken, preventing from errors. This was good part. Now to the bad part. This protocol is not mature enough. For example, you can request list of all open/filled orders for today (this is done via requesting list of executions), but - surprise - this list doesn't have "end" mark. It means, that you never know if you received list of ALL orders, or you should wait for more. There are ways to overcame this issue (namely - you have to keep ALL messages for the day, but if you lost them - you doomed), and I coded for that, but let me tell you guys - it's pain in the #ss. Let me be somewhat technical here. Some brokers provide their own API, so you can link their lib to your program and use their DLL. Here comes "small" issue - so far ALL APIs that I've seen are designed to be used in Windows (that's OK for me), and require you to have Windows message loop running in your program (that's their way of synchronizing/sending messages). Basically, it means that it's pain in the #ss to use their API in a windowless program (like service, for example). I understand why it is done this way - 99% of API users create programs with user interface, so they will have that loop anyway, and who cares about the rest - black box developers?! It has nothing to do with the error handling, just some rant - I have to spend precious time to solve these issues, that could have been totally avoided if programmers in broker's companies would have been a little more intelligent. Anyway, let's move along. Now we have some API to use, and we want to apply our logic. Here is the scenario how human works: 1. (Open a position). Send an order (order A) to buy X shares of Y stock at market (or limit) price. 2. Get filling confirmation. 3. (Protect position). Send a stop order (order B) to sell X shares of Y stock if it fails below Z price. 4. (Close position). When decision is made to close a position, send an update request - to update order B to market. Once it gets updated, it gets executed and position closes. The pitfall is obvious: if something goes wrong between steps 2 and 3 (for example, workstation goes down, Internet conection goes down etc) and stock at the same time plunges 50% down - account gets hit of 50%, because protection order (order B) is not in place. Possible solution for human is to pick up a phone and try to place order by phone. Now, how it should work for blackbox (or carebear human): 1. (Open a position). Send conditional order (order A) to buy X shares of Y stock at market price; once filled, server creates a corresponding stop order (order B) to sell X shares of Y stock if it fails below Z price. Quantity for the second (conditional) order is always equal to the quantity of the first order, so it is not specified in the request. 2. Get filling confirmation for the first order. 3. Get creating confirmation for the second order. 4. (Close position). When decision is made to close a position, send an update request - to update order B to market. Once it gets updated, it gets executed and position closes. As you can see, because creation of the order B takes place at the server (not a client!), position is always protected (unless server itself gets down, but chances are way less). Even if client goes down right after opening a position and never gets any conrifmation, position is still protected, as protection order was created on a server once opening order got filled. Scenario sounds reasonable, right? Wrong! At first, I found this company: MBtrading.com. They provide FIX gateway (good!), they provide complex orders for the step 1 (some limitations apply, though: you can place order A limit + order B stop, but you cannot place order A market + order B stop). But - they cannot provide step 4. Literally: you cannot update order B from stop market to market. You can only update stop market to stop limit (very useful, right?). Then, I found Ameritrade. They provide their own API (some issues exist, but nothing major), they provide complex orders for the step 1 (no limitations, by the way). But again - they cannot provide step 4! To be exact - part of complex order cannot be modified at all. It can be only cancelled. After that, I reverted to the human logic, i.e. no complex orders to start trade, and take a risk between steps 2 and 3. Well, MBTrading still cannot provide step 4. Ameritrade can, but does it very strange way - they cancel order (on a server side) and issue new one (also on a server side) instead of just updating the order. This created some minor technical issue (I've already fixed it), but at least it works. Now, once I don't need complex orders, I started to look for other brokers - may be they got something better - may be better price, or API, or stability. To my surprise: - Genesis: cannot modify order at all - only cancel. - TradeWallStreet (also know as ECNDirect and other names): can modify only number of shares in order, not it's type. I did not check others, but have a feeling, that situation is not better there either. By the way, speaking about error handling. How do you like error message from Ameritrade API: "Your order MIGHT have been sent". Nice, right? While it's not an issue for human, how my program should react to this?! So, my conclusion is this: if you have a blackbox for trading - you have no broker suited for you. All-in-all, I'm still with Ameritrade. They are not ideal, but still no one else is there for me who would be better . And I don't like this situation - there should be at least SOME competition!
Maybe you need to look at higher-end brokers that cater to high-frequency trading. Such as Lime Brokerage, http://www.limebrokerage.com/index.php?title=Professional_Traders
Let me reiterate my statement again: In my price range Which is $5, max $7 per trade (all fees included - even ECNs). As soon as I will have over $100K in the account, I could start using more expensive brokers, but till then - sorry, not everyone have a million to start playing with.
And you would agree that you get what you pay for. Ironically, almost everybody is using FIX. Even Lime Brokerage which caters to high-frequency automated trading is using FIX so clients are faced with the same problems you're faced with. Perhaps what you need to consider is co-locating your black box with the broker to minimize disconnection issues that you are trying to solve.
(Sigh) Yes, I agree. I must consider myself lucky to even have $25K required to begin daytrading. Those lawmaking a$$holes did everything possible to keep poor people enslaved for the rest of their lives. I'm surprised that they didn't set minimum to somewhere around million - to let only rich people have freedom.
I agree with you that there needs to be more competition for "trade handling" protocols. There has to be something better than FIX. But here's the killer... All back-end clearing houses are pretty much standardized on FIX so we probably will never see a better alternative to FIX protocol. Have you considered looking for brokers that support FIX 4.4? http://www.fixprotocol.org/functionality FIX 4.4 has support for Multi-Leg Orders and Trade Confirmations that 4.2 does not have.
MBTrading supports FIX 4.4. And I can create complex orders with them. But I already described the problem with them - I cannot modify existing stop-market order to be just a market. I can change only from stop-market to stop-limit (and vice versa). Or cancel it and post a new order, but this introduces another risk. I can issue update as often as I like without worrying of creating an issue. In worst case this update will be just refused, while if I do cancel/new sequence, then I'm exposed to the risk of doubled/tripled etc oder. For example, consider this: I canceled original stop order, got confirmation. Then I sent new exit order, but did not get confirmation - neither good or bad - just silence. How long should I wait before trying to post it again? And what will happen if first one finally will come through - I will now have TWO exit orders!
Why don't you go to a prop shop like Echo where you can connect via API and get leverage? That's what I do.
That's not his point. His point is that existing trade handling mechanisms need to be improved to strengthen automation capabilities for the retail trader.
Getting the brokerages to allow really tight coupling of a system with the brokerage has been an uphill battle for a long time... same goes for the software developers, some of them that I tried just want you to use market orders and if they get out of synch with your strategy... well, gee, flatten manually and restart the strategy... it's an industry that is in it's infancy, pretty obvious, like car industry in 1910 or computers in 1975... In the 1990's I wanted charting packages to give me daily bars that updated intraday... I would talk to the developers and they just didn't get it... now it's the tight coupling with the account, I talk to developers and they don't think it's worth putting a lot of effort in on it... but the first one that gets it right will be the standard for a long time to come... it's hard to be number one but easy to stay there... IB's API has some bugs in the area of getting some of the account info correctly but I think you can get a list of your orders and their status, and your account values too once you work around the little inconsistencies... I'm not expert yet, just been reading the forums at Ninjatrader and Openquant but I'm headed right where you are going and I don't expect the technology to be what I really want it to be for a while, might as well get used to that if we are innovative...