Dear All, I finally figured out how to setup and run a scalable server with Amazon EC2 (Elastic Compute Cloud). It sounded to me like the ideal platform for an algorithmic strategy since it would be very reliable and could also temporarily boost computational power for an occasional optimization or market scanning procedure. However, based on my testing, it was pretty slow. Big disappointment since it was so challenging to set up. It is definitely not as instantly scalable as Amazon makes it sound- you can fundamentally see why if you setup an instance on your own; basically it takes about 10 minutes to bring more processing power online and it is done manually by booting up a slave server to help the first one. In the data mining application I used as a benchmark it was consistently 25% slower than my tiny laptop over ten iterations. Screenshots and more details are on maxdama.com. Has anyone else tried EC2 or some other CoLo host/virtual server as a system platform and gotten better results? I'd really like an alternative to my laptop with its wireless internet. Regards, Max
I've used the EC2 service many times. I find it to be quite useful in terms of data mining and large data processing. The power of EC2 isn't in using a single instance. It's using multiple instances, in an inexpensive manner. In my case, I had twenty years of historical information on three hundred companies with forty data points for each date. I had to perform calculations on these data points. On my computer, it was scheduled to take over a year. With twenty EC2 instances running, it took a day. So running a one versus one test is pointless. You aren't using EC2 for its intended purposes. Also, once you get the hang of it, switching servers on or off is pretty easy. For immediate load balancing, perhaps it isn't the best solution -- but if you know that during the hours of 1PM and 3PM, your website gets more hits than 2AM and 5AM, you can save yourself some considerable coin by switching on and off extra web servers.
Corey, Thanks for the reply. I was really hoping for good results especially after how long it took to set up so I'm eager to test your suggestions. Three questions: Which AMI are you using? Can you give me some examples of how much the total costs are for some of the processes you were running? Can you point me to a good guide/tutorial that you used? Thanks again for replying because I felt like I wasn't getting the full potential of the service. Your application sounds the same as mine so I look forward to your suggestions. fyi I'm using the ubuntu desktop-enabled ami with elasticfox (basically the simplest possible setup) Regards, Max
I honestly can't remember what I used. It was well over a year ago since I needed it -- and long before they came out with the different scale servers. All I did was create a packaged bundle of tools I needed, then loaded them on my website. When I loaded a new AMI, I would just wget my tools, unzip, and rock & roll. As for tutorials? I was using the Ruby DRb API, so I just looked up DRb EC2 tutorials and found some stuff on google. As for cost? Maybe a couple hundred dollars. Well worth it, considering the alternative (waiting years or buying lots of expensive computers). Sorry I can't be of any more help.
Corey, Recognizing a false belief is always a challenge, you did that for me. I don't mind searching on my own. Advice from someone experienced is really invaluable for becoming oriented in a new idea. I get sentimental when I don't sleep enough fyi. Regards, Max
I didn't end up doing much with EC2. For what it's worth, It's a lot easier to run instances and manage AMI's etc using the firefox addin for EC2, elasticfox: http://developer.amazonwebservices.com/connect/entry.jspa?externalID=609