While I'm shooting my mouth off, I'll add something about exceptions which were being discussed earlier. I'd agree that mostly they should be handled "softly" ie recovery in whatever way possible or just ignoring them may be appropriate in some cases. Most exceptions should probably be logged with with various logging levels. The log should be monitored and alarm generated for log events of sufficient severity (eg send SMS message, flash lights, make noises or whatever). Perhaps shutdown for highest severity. But you need a well thought out logging system with multiple levels.
Yeah. And I'm using ZedGraph for charting right now which is very nice, well built. But it doesn't have any where near all the plethora of options the Java equivalent, JFreeChart. But I'm not sure about the licenses. We do have to keep an eye on over dependence on licenses that won't be compatible with dual licensing later. Again you can post this on the site once I have it up. But I wont' spend more time discussing it right now. It's not the correct timing to do a conversion. But I'm open to doing it in the near future. AGain, what I really would like to do is have an invincible core tick engine that can run strategies and indicators on any of the platforms just by interfacing. One important design characteristic of TickZoom is a clean separation between the engine and the model of custom models. (shorter to type than strategies and indicators) so I will say model from now on. One of my top requirements is that once we're done with Code Review, Alpha, and Beta testing the first time, we need to commit to backwards compatibility from there on at as to the interface to models. That way people who invest work (like myself for example) into building and testing a model won't have to rework that due to future enhancements or ever a change in the core language or technology of the engine to get better speed and reliability, etc. See what I mean, we can't orphan anyone. Unless they all agree, of course, that it's worth it for the new features. It will have to be community driven those decisions. If EVERYONE is on board with converting to java, we can do it sooner. Wayne
Hey, important correction in some numbers I posted earlier. I should never have done that from memory. It turns out I confused bytes for ticks. Anway, erase all that, here's actual numbers. These are as good. I'm curious how they compare to the rest of you? I can get the performance even faster but here goes. I'm using 11.7 Million ticks which is a 159Megabyte file. So that 14 bytes per tick. These are not DOM volume ticks. Just the bid ticks. I have 2 formats. No point in wasting bytes if you're using tick data with volume and dome. Again TickZOOM accepts them interchangeably. It takes 20 seconds to load all these ticks into memory. Check my next post about a question related to the performance. Wayne
if you have not seen it yet, MS recently released a charting control... http://code.msdn.microsoft.com/mschart
I can't give you a good number on the speed of processing ticks in a back test a the moment because... it's taking far longer than yesterday. Yesterday I converted all the bars to using double data type. I originally used only int data type. I haven't committed that code yet so I might back it out. I used int by always multiplying the forex price by a factor so the least sig. digit is the 1/10 of a pip. However, indicators have always generate double since some of my indicators generate fractions. Well yesterday I wanted to compare a "crossover" of an indicator to bars and it was ugly code since they're different types. It requires replicating several methods. However since it's so much slower to process today, I did a benchmark of double verus in and double took 16 TIMES longer to multiply 4 X 7 than with the int data time. So I need to convert the prices back to int s. And run the speed test again. I remember it being about 30 seconds for this 5 years of data or 11.7 million ticks but again I'd rather get a hard, accurate number. So I'll have to find some other way to solve the indicator versus price issue. Sincerely, Wayne
It's a real challenge to write good clean code with performance is so critical. This double and int issue is annoying because I need to use a List<double> and List<int> polymorphically but I can't. Then only way would be to have an interface or abstract class that boxes and in boxes the parameters and accepts both int ant double. But none of that is possible I had it that way before but it kills all the compiler optimizations and destroys performance. This just gives some of your object oriented purists and taste for the reason why some things aren't so intuitive. I'm tried to hide all that complexity in the engine. What bug me is this is something you use alot in custom models potentially. I guess it's alright it will just proliferate the methods needed to calculate Highest() Lowest() Crosses() etc by overloading them all for the different data types. CrossesOver() will have to be defined like this... Code: CrossesOver(List< int >,List< double >) CrossesOver(List< double>,List< int >) CrossesOver(List< double >,List< double >) CrossesOver(List< int >,List<int >) CrossesOver(List< int >,double) CrossesOver(List< double >,int) CrossesOver(double,List< int >) CrossesOver(int,List< double >) CrossesOver(List< int >,int) CrossesOver(List< double >,double) CrossesOver(int,List< int >) CrossesOver(double,List< double >) Now multiply that times the CrossesUnder methods, and then times 2 so you have an optional 3rd arguments of the bars back since the cross over. That's a ton of overloaded methods. However, that won't damage performance like a virtual method or interface would. so there's no way out that I can see. Still the positive aspect is that when using these methods in your code, with all combinations defined. And just use CrossesOver(Bars.High,average,4), for example, which means high of 4 bars crosses the average of 4 bars ago. Sincerely, Wayne
Pippi, I got a lot of additional thoughts about how to handle unlimited about of ticks. BTW, I hit that same wall with NeoTicker. I was reviewing that code and realized several ways to optimize it to double the bandwidth for rolling unlimited ticks through the back testing. The issue with performance is always figuring out where's the "bottle neck" blocking bandwidth. Wayne
FYI, I'm going to become more brief about details in this forum till we get the tickzoom.org site up. They're transferring the domain name now. Then we'll keep everything organized and communicate over there. Still, I plan to have that site link new visitors back to this thread on elitetrader for any "General Questions" rather than on nabble. Nabble will be for the approved users of the software--which will be limited in the beginning. That way, elitetrader will stay happy with us keeping traffic here and it'll keep the exposure to TickZoom. And t will still allow us to better organize the topics and issues for TickZoom. I'm assuming that as long as TickZoom remains free and we link traffic back here, they won't be upset about posting a link here to the tickzoom.org site. Sincerely, Wayne
Folks, Due to a PM I got from a programmer who works in the infrastructure department of a prop shop, I developed a better vision for TickZoom. He pointed out, as I should have assumed, that those institutions have unix/linix based systems that trade with redundant servers sitting close to the exchanges, mostly written in C++. They don't use anything on windows for automated trading system. Just for GUI interfaces. I'm certain those system cost well into the 6 figures to purchase. So now, here comes the full vision for TickZOOM. TickZOOM will become a world class, open source trading platform which rivals the reliability and speed of institutional trading systems but accessible both in price and features to individual investors. That means we'll eventually have to support the execution and quote server on Java as well as C# so it can run on Linux/Unix and windows. That's a smart move because Linux/Unix is a FAR more reliable operating system for this kind of thing. And to boot, it's FAR cheaper to pay for virtual or dedicated server with managed 24/7 tech support at a data center. Clearly that's the direction we're going sooner rather than later. The ONLY reason I didn't do that before was because my broker only has a windows COM interface to accessing their data and my account or else a costly FIX solution. On the plus side, I have become a C# performance tuning expert now. So I need to quickly find a reliable broker with affordable FIX to test and implement this. I want to get this ball rolling. Did I mention that most of my experience comes from building Unix/Linux based high performance infrastructure using Java or C++? NOTE: I just read that db40 is written in both Java and .NET. I'm going to go that route too. They were a successful open source model. Now they apparently test on Mono too. Some people love c# and want to run windows/mono. Others want Java for Windows/Linux. It won't be hard to do both. We'll just have to keep track of tickets for bug fixes and defects and replicate the same fix or feature in the other language each time. We can add a special status to keep track of ticks that have been fixed in both languages or not. That won't be much extra effort. Any suggestions on a broker? Sincerely, Wayne
Hmm. about the double vs int issue. I just discovered C# has some other language and type tricks that I need to learn. I'll have to get a book to read. You have to be careful with features in languages though. Some save code but KILL your performance. Later, on tickzoom.org. I'll post a tutorial of things to watch out for in writing your custom strategies and indicators to keep your performance great. Plus explain how to use the built-in profiling tool to pin point and resolve any issues. Wayne