Do not trade the ES contract until expiration day. The front contract in the ES changes the week before expiration at 9:30 AM EST Thursday morning.
Did you write all new Java code to work with the IB API or did you start with 3rd party Java code? You can also try using ES as the signal basis for your ES trades and see if it gives better or worse results than using SPX.
Thanks for your advice on trading the front contract. As for the code, yes, I have written a trading system from scratch using the Java API. I have been working on it for about a year now, but initially I had a crude system up and running in 2 weeks. I originally started by modifying the small java sample code from IB to suit my own needs. But eventually I re-wrote everything using a more modular and object oriented structure. As you say it is probably worth looking into how using the ES signals will change the algorithm rather than the SPX. But then I suspect it won't be that different, as typically my profitable trades can last sometimes from 15 minutes to even a few hours -- so my algorithm does not really depend on the microstructure of the price movements, but rather on the longer timescale trends. The algorithm triggers entries only at extremely well-defined and specific setups which I have verified through back testing to be likely profitable entries with high probability.
Happy to provide useful information. For the time frame you are using you I agree that you will see little difference between using ES and SPX as the signal data. You might see a difference if your trade time frames were 10 minutes or less. One advantage of using ES as the signal trigger is that you can also trade the overnight market (if that interests you). If you are interested in other ATS Java code for the IB API take a look at JBookTrader. http://code.google.com/p/jbooktrader/
JBookTrader looks very impressive. I will download and check it out at some point. But right now I think I have enough of what I need with my framework. I like the flexibility of my own system since I can do pretty much whatever I want however I want to. My code is split up into a number of manager objects each responsible for it's own thing (e.g., logging, accounting, trading (buying and selling), signaling, watchlist maintenance if trading regular stocks, market data management, database manipulation, and a TWS base object to communicate with TWS,etc). I built a mySQL database on my machine and my framework interacts with the database to do data downloads, cleaning, and back testing. Regarding trading overnight, I have thought about it but so far, I have become quite familiar with the market action during regular trading hours, so I will probably stick to that for a while.
You are my hero! I have recently started doing this myself. I am creating a layer of abstraction around the data feed and the order processing just in case I want to switch broker. I am trying to keep everything lightweight. The overall architecture is quite similar to yours. I may be using an embedded database like HSQLDB, but I have not decided yet. Is back-testing integrated into your framework, or is it separate?
hi Lev, Well it is still a work in progress. The Java code is functional for forward testing and live trading at this point, but I still have a lot of work to do to make it a complete system. You correctly point out two parts of my framework that is still not quite completed, unified, or optimized yet. For the database part, I have chosen mySQL just because I am new to database admin and I thought mySQL was the easiest to learn and get up to speed with. But I have heard that it may not be the optimal choice for trading applications (e.g., I have read some people preferring postgreSQL). But it is a huge improvement over flat files. The performance is quite good as long as the database is indexed and partitioned right. The other part is the back-testing part. Here, I have chosen not to build it into my Java framework, but rather, I wrote a separate Python class library to do backtesting and plotting (e.g., PL curves, daily entries and exits, etc). Originally I was using the R statistical package to do all my back testing, but there was another thing that I badly needed in my system, and that is the ability to monitor my system while I was away from my machine. So I set up a webserver and I wrote a python matplotlib class library in conjunction with a set of web pages written with Google Web Toolkit to display my running forward tests (and live runs) while away. While I was doing that, I shifted all my back testing to Python as well using Numpy and matplotlib charting so that I am now using my web interface even while I'm working on my machine. This python library has a database class that queries the same mySQL database that my Java framework updates. The only reason my Java framework deals with the database at all is because I run my Java forward test consistently every single day -- so I built in an automated downloader in the Java code to download that day's historical data after market close automatically. So I don't ever have to deal with downloading again. I also have a windows cron job that runs and cleans up the data and runs my most recent back test algorithm on that day. This allows me to compare the forward test (from IB TWS paper trading), and my back test. I can visualize every trade, the time and the price of execution, and that way I can make sure that my back tests are an accurate reflection of my forward tests (e.g., slippage and commission). By the way I also went live several times purely for software testing purposes, and I have been quite impressed with how well the performance matches up with the paper account. So my system is not one monolithic package in Java, but has various components in Java, Python, and a webserver to monitor performance of both back tests and forward tests. How about you? Are you writing everything in Java?
I am building mine with Scala because I can write more succinct code with its functional features. It also encourages the use of immutable data structures. Overall I feel I can write safer concurrent code with Scala as opposed to Java. I am going to unify my trading and back-testing framework though. I will just generate ticks/bars from historical data and feed it through my framework. I have used both PostgreSQL and MySQL. I somehow have the impression that MySQL is easier to work with. You are right that having optimized queries and indexes is more important. I will probably use a database for data storage, but I am going to do most analysis in memory. It is amazing how cheap memory is and any call to an SQL database will have some overhead. Certain operations are difficult to perform efficient in a database. I am not planning on having much of a GUI beyond a few buttons and statistics. It is interesting that you are using a web server. Nowadays it is much easier to build a rich UI using web technologies. GWT is wonder (at least for Java programmers)! Are you planning on monitoring/controlling the live environment from a web interface? BTW, does your framework handle situations like partial fills? (e.g. you send an order to buy 5 contracts, then you get fill on 3 contracts with 2 remaining, now you system will to stop and reverse and go short on 5 contracts.)
Okay excuse my ignorance, but I just looked up scala and it seems like a really nice language. I will probably take a look further and see if it has a place in my framework. Regarding databases, one of the reasons I built the database was because I originally was downloading all the SP500 stock minute data, and it was becoming extremely painful to maintain the data using just flat files. Now that my algorithm is just using SPX, SSO, and ES, I admit it might be overkill. But it is a huge relief that I no longer have to worry about corrupt files and fixing the files and missing data, missing symbols, missing dates, missing minute bars, etc etc etc. I used to spend hours a week just maintaining and cleaning the data, changing the file naming conventions, re-downloading missing data, etc. Now I hardly spend time thinking about the data -- it's just there, and it is cleaned, and it is automatically updated every day. Regarding the web server, right now I have built the bare minimum necessary control -- that is to stop and liquidate everything. So I have the peace of mind knowing that I can stop it at any time. I can also view the log from work, and see the real time plot of the P&L as well as the signals that it is triggering on. Beyond that, I am not sure if I would be needing any more fine control. Of course there is the age old debate about whether manual intervention is beneficial or detrimental to ATS performance. Regarding partial fills, I only use market orders, so I never have an unfilled situation. I do see partial fills as the order gets filled and I do log those as they execute, but my code does not acknowledge the trade is completed until it is filled. But I do change my account balances while the partial fills are added -- this is because I like to have an accurate reflection of the amount of cash in my account at all times so that I can manage position sizes for new buys and sells if they happen while partial fills are occurring.