I am considering moving my automated trading system (using IB Java API) to Amazon EC2 to lower my latency and / or improve connectivity (I live in South Africa) and to be able to travel light when I feel the need. I am a newbie with Amazon EC2 and Linux. Why Linux then? Mostly costs, and secondly I believe you need less resources to run a Java IB API efficiently on Linux vs Windows. My problem is how do I log into IB (Gateway or TWS) without being able to log in remotely? How will I be able to see what security codes to enter (from my security card). Is there a way to remote desktop to a Linux server, or is their a way to retrieve the requested security codes in the Java API? Another possibility is to install the Ubuntu Desktop version on Amazon EC2 to be able to remote desktop from wherever. I got this idea from maxdama's website. I don't know if this is very inefficient. How do you use IB in conjunction with Amazon EC2 (or any other server solution)?
You can remotely access Linux by creating a ssh session using password or public key authentication (best) and then tunnel vnc back to your computer. This way you can securely access the desktop of the computer you are managing with full access to the GUI and everything is running over an encrypted tunnel. This will allow you to enter the codes when TWS first starts up. You would probably want to add IBController as well to keep TWS running continuously. PM me if you need any specific help with this. I don't have any experience with Amazon EC2 yet. Sounds like a pretty cool service though. Good luck.
First, you will need a xWindow (graphics interface for Linux) installed in order to run TWS or IB Gateway because Java requires it. You can install it on the server version of Linux but you can as well go with a desktop version. If you have xWindow installed you can either use one of many popular VNC applications to establish remote connection or (if you have Lynux at home/in the office) you can use your machine as xWindow server (and possibly route connection via ssh). Second, via account management you can obtain a second login for your account. You can configure it so that it doesn't have a security device and is permissioned to trade but not to withdraw funds. Or you can request that your current login works without a security device. Without a security device, there are a few applications that will type in the login name / password automatically whenever TWS or IB gateway starts. With 2 accounts you can run a second TWS on your dektop in order to monitor trades, orders and the state of the account.
Thanks brocklanders and Leed! Valuable advice indeed. Would you say that installing Ubuntu Desktop on Amazon EC2 is less efficient (resource hungry) than installing xWindow on the server edition? Is there abig difference between the two paths?
I'd be curious how cost effective running from the Amazon cloud would be vs. the costs of running a dedicated (remotely accessible) home computer. The market data feed is what worries me (consumption of cloud bandwidth). Looks like it's $0.10 per GB for data inbound to your cloud instance... maybe that's not too bad? It would depend on how many symbols you subscribe to and whether you're getting tick data vs. X-minute bars, etc. The cloud would certainly have better longevity (ie the chances of your home computer crashing mid-day would be higher vs. what you'd have in the cloud). Interesting idea...
I'd be interested in knowing the costs as well if you don't mind sharing some specific numbers. As far as server vs desktop, I would imagine that you can do everything you need on Ubuntu desktop and it should run just fine. I run TWS on Ubuntu desktop and the performance is great. However, I run into problems with some Windows specific chart software that just doesn't work well using the Wine emulator. If you have any software that relies on .NET you would be better off going with a Windows server than Linux just to save yourself the headache. Please keep us posted on your progress. This sounds like an interesting project!
No specific numbers from me, but I remember several data vendors were working on building Amazon cloud-smart access. Remember that latency as well as bandwidth matters. For HFT or even any trading that justifies a collocated server, don't you want guaranteed resources? One would hate to be sharing a host with WikiLeaks or something. From my days of using shared websites hosting, we had alot of outages due to the machines getting hacked. Maybe that has been fixed in the cloud, but it is always easier to spread to the next machine (hosted or virtual) if you have fast connectivity.
Typical real data bandwidth is tiny... especially if you are not tracking hundreds of symbols with market depth. For trading a dozen of actively trade futures you will likely fall into below 100mb per month... unless you frequently load tick-by-tick history and the like.
I would typically take snapshot data every 5 secs for a maximum of 20 symbols. I agree with LeeD, I don't think the bandwidth (costs) will be a problem. Your costs will mostly depend on the size of the instance and the total time that you are running the sytem on EC2. I roughly calculated that it would cost just over $80 per month if you use a high CPU instance (2 virtual cores and 2.5 EC2 Compute units each) with 1.7GB memory. That is if you assume that the system is running 23 hours 5 days a week. The cost would be half if you use a small instance (same memory but only one virtual core). The biggest advantage of going this route is the connectivity and latency, definately not the computing power. But IMO that is exactly the weak point of my automated trading system, especialy if I run a DSL line from South Africa. Look at the latency advantage at Amazon EC2 to IB: http://www.maxdama.com/?p=207
I've been playing around with EC2 tonight and have a Windows and Ubuntu w/ X Windows up and running. The Windows instance was much easier to get working with TWS but Linux is doable once you know what you need to install. I started with a public AMI from Canonical running 32-bit 10.10 Maverick Meerkat. The AMI id is ami-ccf405a5. Next you install gnome desktop and VNC server using the following command once you are at a shell: sudo apt-get install ubuntu-desktop tightvncserver Next you will need to install java: sudo apt-get install default-jre Restart the instance and then setup vncserver and install a vnc client on your machine. puTTY is a good windows client and there are lots of instructions on how to get the vnc session tunneled over ssh. The Windows install path was much easier than this and I will probably just use it for my testing. I just did the Linux instance to see if it would work . Some are also recommending using NX server instead of VNC these days. I've used NX server on my home Linux box and it is very fast even when compared with tightvncserver. You might look into that too if you get very heavy into using Linux remotely.