Folks, Please join in a discussion of your thoughts and/or encouragement to release an open source black box trading platform in C#. This is a platform that I developed and use myself in automated trading. I'm considering releasing it as an open source project for others to use. I like the idea of a community collaborating on building a great system. Unfortunately, there's no such open source system available today. Let's call this platform "TickZoom" for the sake of clarity in the discussion. Why am I reluctant to release it? Well, very few will ever use it unless there's very good support. So I've been exploring ways to generate revenue to provide support while still making the system freely available and open source. After researching more about successful business models for open source, it works best to always keep the source code free of charge and easily accessible so the community can contribute without charge. To pay for full time developers, an organization sells services like creating a packaged product that's easy to install and selling support contracts, training, documentation, etc. In other words, some people can contribute with time to the project while others contribute with money. But the software license to use and copy the source code or software will remain free of charge. In my next post, I'll attempt to explain some of the features of this platform and how it differs from the commercial close source ones on the market. Sincerely, Wayne Bold = text edited on 20081209 17:30
Why another trading platform? Well I previously back tested trading strategies on many other commercial products. In all cases, I ran into 4 major problems. Plus a list of features I would have liked weren't there. The several major problems of other systems: 1. The systems either didn't support tick data or their performance was painfully slow at back testing on tick data. 2. The systems didn't offer any truly hands-free way to run a strategy so that it stays connected and running 24 hours a day. 3. Any systems that claim to do those things cost insanely large amounts of money and would deplete my bank roll to use for trading. 4. Worst of all, all those systems I tried were closed source which means they don't allow users to make enhancements and contribute them to the project. In most cases, they didn't have a good debugging environment to find and fix errors in trading rules either. So what does TickZoom do differently? 1. TickZoom is built from the ground up to back test and run purely on tick data. All bars in the system like second bars, minute bars, hour bars, tick bars, range bars, and volume bars, always update for every single tick change. Longer term bars like week, month and year update every minute. That means your trading rules see the data and bars exactly as in a live trading environment. My personal strategy that is running while I type this email continues to perform exactly like it was proven to perform in back testing. 2. The ZOOM in TickZoom means that the programming architecture and logic has been extensively optimized for speed at processing tick data. It also uses multi-cpus if available. To give you an idea of the speed, the system can run a combination intraday trend and mean reversion back test on 5 years of tick data (115 Million ticks!) for forex in 45 seconds and only uses 500Meg of memory. In fact, it only takes 15 seconds to load all the data into memory. And the system conserves memory too. That test takes less than 500 megabytes. That kind of speed actually makes it possible to run optimizations on years of tick data. On a quad core CPU, for example, you could run 100 tests in under 20 minutes. 3. Complete time frame freedom. Most platforms make it difficult to use a variety of bars in the same strategy. However, it's often useful to compare hourly bars to daily bars to tick bars and even month or week bars in the same strategy. In TickZoom, you can use as many different time frames as you like simultaneously mixed any way you like. For example you can have a strategy that uses constant range bars but also refers to indicators that use a combination of 5 minute and hour bars. You may have exits or stops that refer to sessions, daily or minute bars or trailing stops, etc. etc. In other words, TickZoom offers complete time frame freedom. The test mentioned above uses no less that 12 different types of bars. 4. TickZoom places total focus on black box, automated trading. This means for example, that the charting doesn't support manually using a mouse to draw lines, add indicators, etc. Instead you use a simple API or components to programmitically draw those indicators. However, the system does support live, manual trading. The reason it works this way is because after you have your indicators defined in your strategy code and tested it, you're ready to run your strategy live on the TickZoom black box server. You "just flip a switch" so to speak and it run without any charting attached. There's a lot more features. But these give a good example of what makes TickZoom different. Sincerely, Wayne
the speed.... how do you store the tick data which is an issue ? i saved my tick data as simple as txt files. also importing e.l. scrtips is something needed b/c most codes are wrriten with it i am not a programmer, just a trader and i hope that it will be easy enough for me as well great work!
Backtesting with tick data is highly desirable, but it's a tough ask for strategies involving quite a number of instruments eg baskets of stocks. First you have to get hold of the tick data for all the instruments, then store it and process it. For the small trader it may not be practical. So I reckon you also need to support backtesting with ordinary bar data as well.
Text files are quite poor for this sort of thing. Just converting the text to binary floating point chews up quite a bit of CPU when you are talking large amounts of data.
Thanks for the questions! I'll answer one question at time. First about tick data format. Format. I use a different format for performance. Text files take a lot longer to load 119 million ticks. Instead of 15 seconds, that could take several minutes every time you run a test. So the platform uses tick in binary format. NOTE: I currently collect tick data from MB Trading on 12 major pairs and will eventually make that available. My tick data has both Bid, Ask, Volume (synthetic), and 4 levels of DOM and separate ticks for any change of DOM or bid/ask information. Unfortunately, that data only goes back a several weeks. But it's MOST enlightening to see the DOM and volume changes on Forex when news happens, etc. Basically, it's a lot of data but I use an optimized binary format for super fast loading. Those 119 million ticks only takes 159 Meg file. Of course, with my expanded tick data, it takes about that many ticks per week. (NOTE: the processing doesn't take nearly as long due to optimization.) I do have a converter to convert from a text format. But it will probably need to be more flexible to support different formats. Sincerely, Wayne Walter
Time-Series databases are a highly specialised field with two major players: http://www.kx.com/ http://www.vhayu.com/
I'm glad you mentioned E.L. I'm considering a couple of different approaches to that. The best, I think, would be to have various examples of how to convert an E.L. script to C#. I have already made that pretty easy because I figured out was to make C# provide most of the features of E.L. For example, automatically handling maxbarsback, etc. In fact, I personally had some E.L. code that I converted to C# like pivot point calculations and it was most straight forward to do. The most important thing will be to get all the most common indicators in the E.L. converted so that E.L. scripts that use them have them. The only way to get there is from here though. So we can do it as beta testers or others need one, just convert it and add it to the "pile". Unfortunately, any way I can think of to use E.L. "as is" would either be a big performance hit when running on tick data. Or else be hugely disadvantageous in other ways. One of the nice think I like about C# over E.L. is the use of a debugger to step line by line through the code. I even have a feature to set a bar number to break at and then step through the code for that bar. The debugger is freely available. I use SharpDevelop but Microsoft has the VisualStudio which is excellent but costly. In other words, there's tons more tools to speed development using C# directly. Sincerely, Wayne
That's a very interesting point. My platform certainly has the feature of working with "baskets" of instruments. It even has the ability of working with "baskets" of separate strategies a.k.a trading models on the same instrument, in parrallel. Hmm, that's a fascinating point about bar data. Well, the system already does that in a way because it optimizes and uses 1 minute bars to update the daily, weekly, monthly, and yearly bars. It could be slightly modified to use bar data as another option as the minimum bar size and then fabricate any higher order bars. What bar size do you have in mind? I assume 1 minute data is the most readily available. Or is 10 second stock data easy to get? Wayne
Quite true! In fact databases are out of the questions too. I experimented with several and personally have 18 years experience with high performance databases. Nothing comes close in speed to reading them straight from disk in binary format. - Wayne