Until you are trading real money, you have no position on the CME order book, so you are being filled on some rule set up on the paper trading. It will not be the same.
I mean massive infrastructure is not scary, in fact that shows that the HFT firms are slow. A massive infrastructure often shows a lack of programming knowledge, a powerful infrastructure is a small. Perhaps only 1 computer with a FPGA nic or DPPK/similar to pull packet frames and process them immediately. This method will give you the required trading latency (under 1000 nanoseconds to receive, proc and reply a packet). If you have any kind of infrastructure, by the time the packet propagates through your infrastructure you already at like 10,000 nano+ Now a FPGA nic is like $1000+, DPPK is free but requires a $500+ nic. My teams programming time is free in terms of spending a few moments here and there to work on something that has potential. A spot in the datacenter near the exchange I can imagine would be at least 300$ a mon (colo), a spot that apparently is the closest to the exchange I think go for $10,000 a mon around. A data subscription for data directly from the exchange. I dont know the price on this one. A server (no nic) would be like $2000 tops. You can fit in a $1000 budget easy. $10,000-20,000 if you go all out getting 32~ phys cores 32~ vcores, 160gb+ ram, raid0 pcie ssds. Tho depending on the workload the 10-20k system might be SLOWER than a 1k system. You can get about 1 op per 1 Ghz per nanosecond, a modern high end server processor is around 3.6Ghz. A $300 overclocked older process can run at 6 with record 8Ghz clock speeds. It will be 2-3x faster! Also of course you should use a 32bit architecture to squeeze out every bit of latency. Again depends on the workload. You have a per process memory limit with 32 bit so if your using some crazy in memory lookup tables those will have to be coded to keep that in mind. And that is basically your cost breakdown. It is NOT that much. It is not the kind of investment that warrants multimillion dollar returns. Hell no. Go buy a store chain, spend a few million and make a few hundred thousand. Here its like spend under a hundred thousand and make a few million.
If you know where to get REAL colo for $300, please share! I'm in somewhat agreement with @i960 . If you're trying to win purely on speed, forget about it. That is a game that is all about expensive infrastructure that is unattainable for just about all of us. I think you can still be successful "scalping ticks" in an automated fashion, but there has to be real intelligence behind your entry/exit signals. It can't be all about speed and queue position. If you're only trading futures, I'd ditch IB.... Checkout one of Rithmic's broker's....you might save yourself a penny, and get faster fills. I'm not currently using them, but will be...hopefully soon.
The real issue w/r/t speed and cost is not the cost of physical infrastructure as you've alluded to above, it's the cost of services and payroll. If you're actually going to be competitive in the speed game, it's very difficult to do so as a one man shop. Systems people are quite expensive, especially those with experience in this space. There are ways to keep things cheap and get better speed, but you need to use the proper infrastructure and service providers on the right exchanges. W/r/t position in Q, for some exchanges you can know this explicitly with the correct data feed, however for others you must estimate it. Estimating Q position is a complex problem that is easiest to solve if you have fill data or are clever enough to work around it, which can be done well on some exchanges and products. Start by understanding the mechanics of the exchange and the data feed. Do some thinking about scenarios where you can (or cannot) explicitly know an order came from in front of (or behind) you in the Q. If you can figure out the fill data issue, you can start to develop statistical models for the conditional expectation of your Q position and go from there. The big problems you'll face are as follows: 1. If you don't have your own fill data, you'll either need to figure out how to use the information in the feed or find some in order to test the efficacy of your model. 2. You'll need to deal with deaggregating a price level aggregated feed most likely. Any grouping or aggregation of orders makes this impossible. 3. Certain products, CL for example, do a lot of one lots. Any non-uniquely sized quote destroys a lot of assumptions about Q position in an aggregate book, but too many one lot offers getting hit by other small orders makes it difficult to match against the order of updates to the price level you see. 4. If you're competing for Q position, you need to know that you're going against fast guys. Without the proper timestamps that no commercial historical data sets I'm aware of provide, you will need to make assumptions about whose order arrived first. In practice, unless you're already committed to writing a lot of code and building a pretty robust system, it's not even worth worrying about past just the pessimistic upper bound of SizeWhenIJoined - SizeTradedAgainstLevelSince.
$300 is low. And there's no point using FPGA if you are just 'near the exchange' For a single server at the exchange (CME) you can pay $1000-$2000. That would give you the same access as your competitors. There are more significant competitive barriers other than infrastructure. 1) For your FCM to allow direct market access you need $500k-$1000k in cash. 2) you might skate by on an individual membership for a bit but you open yourself up to disciplinary action. To do it properly you need a CME membership. Sure, there are other types of memberships but they are not fee competitive. So that's another $2M cash + $2M loan. 3) until you have a track record of doing 'enough' volume - about 100k contracts per day - your clearing fees will probably not be competitive. 4). Fast lines between exchanges are expensive. And you will never have the fastest. So people responding to external events will always beat you. All these problems can be creatively solved (except 4) but then you'd still be among the slower FPGA players. With a basic strategy perhaps profitable but not earth shattering. Could be enough to live on and slowly bootstrap yourself to success.
I am not too familiar on colo prices but I often rent dedicated servers, I currently pay on west coast under $100 for 80gb of ram 12 phys cores 12 vcore and 4tb of harddrive space with 30TB a month (100mbps unmetered) on a 1gbps nic. I know colos with electricty for 1U end up like $80-200~ without the bandwidth. I guess if you get a cabinet or half rack itl run you 800-2000 but really you just need 1 server in there with your custom nic. It seems unrealistic to do HFT with serious data crunching involved, any calculation beyond a basic one would be way too expensive in terms of response time. At that point your not HFT but just algo trading, your trading in the milliseconds not nanoseconds. My 2cents on the distinction between HFT and algo trading. This is really informative thanks. This actually seems quite reachable and attainable. The main kind of peeve with this is the barrier to entry, so once you get all this, THEN you can start development. Up until you have all that setup the world of what is going through that wire coming from the exchange is a mystery. (#3) Seems like it can be worked up to. (#4) Seems like that small group of insiders that are everywhere really. You don't really care about beating them.
As rmorse stated, cancel/replace puts you at the back of the line. It is important to understand how the trading platform you use handles this. When a user changes the quantity of an order, NinjaTrader is smart enough to automatically create additional order(s) thus preserving your queue position of your original order(s).