So, let's say I wanted to follow your recommendation and run a big instance from time to time for back testing, how long would it take, in your opinion, to configure ports, setting configurations of the server, uploading the backtesting platform, hooking up to the changing IPs (they change all the time once you start a new instance, stopping instances does not count because you would be charged, only deleting the instance stops being charged). Then you want to install the development environment given your backtesting involves code changes, you need to install certain updates (given you run on a win instance),..., I could include another 10 items but you get the point. My issue is that as long as users get charged for "stopped" instances the whole idea of cloud computing and "dynamic" expansion/contraction of grids makes zero sense. As long as the instance environment gets completely wiped out just because one does not want to be charged 24/7 there is hardly any reason to think of such instances in the first place. But they surely make for a great gizmo experience and you can brag on your blogs that you have been running your testing environment "all the way up in the cloud".
I think the cloud may have its place in trading but purely for signal generation/distribution and execution. There is really no need to backtest in the cloud because you can do this easily and reliably on a desktop machine. To me, the benefit of the cloud is power redundancy and a direct connection to the internet backbone. It's not processing power or storage or scalability. It's knowing your not going to miss a trade because a car crashed into a power pole at the end of your street.
Most of the configuration work can be done offline. Initially it costs time to figure out what changes you have to make. Some changes are only made once after a fresh install of your operating system. Some changes are made every reboot, some changes are even made during running the system. I argue that most of the changes you need to make are fairly easy to automate. Both Win/Linux allow unattended installations. These can be prepared offline, latest updates can be included. Rest of the changes are automated with batch/shell scripts. Amazons API helps a great deal with all sorts of automation tasks you want to do. At Amazon you can use EBS to store your installation. There is no real cost to this. Still all this automation doesn't make it that you can scale instantly. It takes a while before your instance is ready to use. The distributed aspect, reliability and replication of your data, and how you divide functionality over multiple instances in a efficient is likely to give way-way more headaches. (btw. I'm not advocating cloud over any other solution, just saying that the difficulties lay more in the technical than management aspect)
I dont quite understand: a) power redundency? You dont get that with a standard dedicated server? b) direct connection to internet backbone? See a) c) Hugely disagree with you that cloud computing is not about processing power and storage capabilities. Its ALL about processing power and ALL about storage capabilities. Not just in non financial application such as Apple's cloud or Google cloud enabled services. And for me its quite the opposite from what you stated. I would want to run live trading on as little machines as possible given there is no starvation of resources. The more hops you have the higher the latency and lower the throughput. Plus most importantly, the more potential for things to go wrong and to recover. What you really want to use a cloud for is to run some massively distributed testing algorithms and ways to compute complicated functions in parallel fashion for back testing purposes. So I could not disagree with you more.
Even the best cloud servers have outages in car-crashing-into-a-power-pole scenario's. You put way too many trust into Microsoft/Amazon/etc. if you believe such a thing. This is the same as with your VPS host, or your home computer. They do however power you to build a more reliable infrastructure with quite amazing efficiency.
The reference to power redundancy and connection to the internet backbone are my personal requirements for using either a cloud or dedicated server solution. I get a power surge or power failure about once a month that can cause my computer to crash. My DSL connection drops out about once a month also. These both present problems for ensuring I receive my trading entries and exits. Sure, plenty of people use these services for storage and processing power but they are not a requirement for me personally. If I was going to backtest or data mine, I'd just do it on my desktop machine.