I am starting to work on some strategies that would be trading markets in a fairly fast paced fashion 24/7. This means that my previous strategy of 'human back-up' will no longer work. I am in need of some serious advice, preferably from people who have experience with hardware systems for automated strategies. My concerns are primarily electricity and connectivity. Basically, if a lightning storm knocks out my system and I have a few trades on, I am in big trouble -- especially if the system reboots and doesn't know what its current trades are or what to do with them! The same goes for if the internet connection goes down. I know a lot of people get redundant systems (UPS at the low end, generators at the extreme end) and back-up internet connections (multiple companies, satellite back-up, et cetera). But one of my biggest concerns is what to do with systems that go down when trades are in place? Also, even though all these backups exist for electricity and internet connection, there is still big 'geography' risk (i.e. my whole town lost power for two hours the other day). Finally, if I begin running multiple strategies, I am concerned about eating up my time just doing IT work! So I started looking into some solutions, and thought that using Amazon's EC2 and other cloud solutions might work best. Considering most systems are outdated in two years anyway, I can get a low-end system on Amazon for a year for $325, and a high-end system for $650. For a 3 year term, I can get a low-end system for $500 and a high-end system for $1000. Those are pretty damn good prices, and I get to ignore all IT problems. My biggest concern still remains on active trades when connectivity goes down. Does it make sense to develop a back-up system on another cloud (google or microsoft perhaps?) and create a log of all trades on the other cloud? Then if anything goes down, the back-up cloud system would just cancel all active orders. For serious worst case scenarios, I could have a back-up to the back-up cloud. Does this seem like a reasonable solution? Using cloud computing seems like an attractive option. Thoughts?
distributed computing at multiple locations, as you concluded, is the solution. i've never heard of anyone doing this on an EC2 cloud. typically this type of application is farmed out to a datacenter... colo/hosting shops rent out their own machines (assembling them to spec), or you send them your own. yes, you circumvent IT, but you'll be swapping that for network admin and interprocess/inter-server communication. failovers on multiple machines aren't trivial. good luck.
I don't know if you necessarily are swapping IT costs for network admin and interprocess/inter-server communication. A year ago, maybe. But EC2 now allows you to use static IP endpoints, multiple locations, and persistent storage. Plus, since you use Xen machine images, you can also upgrade your hardware to a more powerful machine at any point. And if the time comes, you can also run the machine image on your own hardware (using Eucalyptus). Basically, I think solutions like EC2, and to a lesser extent Google App Engine, Microsoft Azure, and GoGrid.com, allow you to get all the benefits of professional server hosting without the IT costs... I just can't figure out a good method to do the failover. I think I would do a dedicated server in the U.S. and Europe, with the failover being Europe -- which would simply cancel all orders if the U.S. box goes out of touch. But I dunno...
I am also working on recovery, logging all input and read it back on startup should do it. Not that big of a deal I guess.
right, that was my point. syncing your apps to have seamless failover is tough. msg layer, connection monitors, client/server setups, security... all this will replace IT. EC2 doesn't make those things go away, and if you don't have those layers integrated into your framework, trust me when i say, it takes a beat to do... all this also has to be maintained. i did exactly this for the reasons you mentioned. ime, IT spent at home was replaced and exceeded with the time servicing this but ymmv. the cloud setup may ease the work a bit, but 90% of it in my case was at the framework level.
Don't forget that you still pay hourly after you pay $325 or $650 per year -- you just pay a lower hourly rate. Elasticfox makes managing instances much less of a headache.
A good point -- but it is still an incredible discount. Comparing small linux instances, it is about $870 versus $490 per year, and that is if you renew yearly. Even cheaper if you run it out three years ($2600 vs $1100). My point being, $1100 ain't a bad deal, especially since you don't have to handle the IT headaches that come along with hosting your own server.
Make sure that you understand the drawbacks of operating in the cloud, specifically latency issues. You're also giving up a lot of control, especially at the hardware level. After a lot of experimentation, I've settled on renting a few boxes at two datacenter locations, so I have local redundancy (main server/failover) and geographical redundancy provided by an identical mirrored setup at a different datacenter.
Just out of interest (if you don't mind me asking) how are you handling the failover? Is the failover running in full synch with the primary or are you only bringing it online when required?
I have started to look into the AMZN EC2 and S3 platforms. Much of my code is already distributed and componentized and it would very simple to put into Web services. One thing that is initially annoying is that you can only do Reserved Instances on Linux I am 95 delta to be using the AMZN cloud to do some high frequency trading. I will give a review when I have more experience with it.