"And referencing Bars.High[5] means a the high 5 bars ago." It's gotta be said - you'll confuse the crap out of everybody with this. I think it was Gosling that replied to criticism of the omission of operator overloading from Java, saying that you can't write a manual page for "+". I'm afraid I think he got it right. Of course Java does break this rule with concatenation of strings, so it might be a case of do as I say not as I do.
You know, I agree totally with both points. I remember the struggle to figure out which platforms support which brokers myself. I'm assuming we'll do the same thing as Ninja for the paid customers by giving a free version till you want to run it live with a broker. But that only applies to people who want to get a convenient build and install package with support. The source code will always be free to down load. The idea is to always make it free for those who are willing to download and build the system from source code. It's because those people are most likely to contribute ideas and code back to the project. Sincerely, Wayne
Wayne, great initiative. On the programmer vs retail question: I suggest the open-source initiative to be for programmers by programmers. A great class architecture, tick & bar databases, backtesting environment, execution platform and decent documentation. To generate revenue, you could create an openquant or ninjatrader like overlay application on top of this framework and either sell it as an application, or openly distribute in binary (or even source) and sell services for it. I use a combination of OpenQuant and custom C# applications written to the InteractiveBrokers API for execution. I have been slowly building my own framework as I need new functionality. Judging from the TWSAPI forum, I think there are similar people out there. I would enjoy putting the same effort into an opensource framework, and in return getting the fruits of a larger community's labor. I hope to see your alpha/beta release soon.
Are you trying to make all the retail people happy? or all the professionals happy? There's a difference. A major difference from what I'm reading here... If you want this project to go somewhere, I would ignore everything mentioned in here except for the tick data. Also, separate the tester with the execution platform. Have a common lib that will work with both+ ends of the app. Execution should be FIX. No broker APIs and all the other BS. If you put them in, you'll be dealing with all the non-programmers or brain-less system trader. Credibility and feedback comes from professionals using them, not from leeching newbs. Think about it, something becomes a genuine product when the professional crowd backs the project like Eclipse. (You can convert QuickFIX to C# ... shouldn't be much of an hastle and I know that there's a high demand for a pure .NET FIX engine in the industry because there's a lot more front end programmers than the system infrastructure people, and people want a front-end FIX capability... you'll be surprised... seriously) Other than that... What some of us do is to separate test/demo/live using the compilation like DEBUG or RELEASE: #if TEST using x.y.Test; #elif DEMO using x.y.Demo; #elif LIVE using x.y.Live; //... your code for the model like: Order.BuyAtMarket(...) {...} Where Test.Order, Demrder, Live.Order exists... Also you may want to contact: http://www.qlnet.org/ I know a lot of Quants using QuantLib as their guide to code their test on MatLab or SAS. Having QuantLib on C# or pure .NET would get some following with the Quants. Good luck. I'm looking forward to this happening.
Extracts from db4o Documentation : "In simplest cases db4o handles schema changes automatically: * If you add a new field, db4o automatically starts storing the new data. Older instances of your stored class (from before the field was added) are still loaded, but the new field is set to its default value, or null. * If you remove a field, db4o ignores the stored value when activating instances of your class. The stored value is not removed from the database until the next Defragment, and is still accessible via the StoredClass/StoredField API. * If you add an interface to be implemented by your stored classes, db4o automatically starts using it and you are able to retrieve your saved data using new interface. " There is also a Refactoring API to handle more complex cases. To sum up, I have already used db4o in a professional environment, with decent volumes and excellent results. It is one of the best open source technology (with an active community, you get answers in the forums, from the developers themselves) I have come across. For your info, I am about to try to store tick data for the ES for several years in a db4o database, and try to see what kind of performance we get on time based queries. PM me or email me if you want an update on the subject.
OMG, dareminator. That was a lovely post. It made me feel all mushy inside because you express EXACTLY my sentiments better than I did. Frankly, I wish someone else already had this project going because I also want to benefit from the collective contributions. As far as your recommendation for the structure, that also is exciting because that's exactly how it works now. Specifically, I have been careful to keep a very clean object structure for the engine itself. (It can always be improved, I'm sure.) And the GUI part is separate and very simple right now. It only has a form with the date range you want to run, optional replay speed, and buttons to run either a back test, optimization, or live. Finally, it has a progress bar and a log message window. Most log messages go to a log file but some important ones go to the app window. Hey. Tonight I'll post a few screen shots. Depending on which button you push it may pop up one multiple chart windows to view. It writes all statistics to html format, a separate one for each strategy if you're running a portfolio. So the sum total of the GUI right now is 2 classes. Form1.cs and the Chart.cs The system is even broken into separate assemblies. The GUI which I hope will grow over time is one assembly. The TickZoom engine, I call it, is another. Oh, here's an example, right now you have to set optimization parameters in code, programmatically. There's no GUI for that yet. As far as the GUI goes, I've been only adding features if it saves me some time over changing it in the code. But eventually for non-programmers, we'll have to make all this into a nice GUI. And maybe you're right, those can be the services that make this commercially viable. It sounds like you're going one step further in suggesting that the programmer source-code-only version never has a fancy GUI--only the commercial one does. That might make sense if the community is okay with that. And I see your point as a programmer. I don't feel the need for a full blown GUI. Just a powerful library is nice. Hmmm... that seems like a very clever model. Does anyone disagree? What was exciting mostly about your post is that we have the same vision for the community of programmers. It will only take a handful of us with this kind of vision to start a snowball effect that grows into something truly wonderful. It's seems the key at the moment is to form all this conversation into the "roadmap" for the project. I'll post a draft shortly. That way we can tweak and get it clear very everyone. Hey maybe after we get this going and prove it can be financially successful too, someone else or a company may agree to take it over. We just need to make sure that we legally protect the code we contribute to make sure it is always open source and free. The model I like for keeping the code public is to form an LLC which owns the LGPL license. Then any committers to the project get added as members to the LLC so their vote counts in any decisions related to the source code. That assures it will always stay open and free. Sincerely, Wayne
Good point. By the way, the reason I love tick data is that it gives extremely accurate idea of the execution of my orders during testing. To retail people, have you ever had strategy that works well in testing but fails in production? I found most of the time it was because of testing on bar data. When I switched to tick, I got the results in production that match the testing. That's exciting because that's how TickZoom works now. Got it, I'm sold. I feel convinced that FIX should be the next interface to tackle. But I want to eventually support brokers also as a secondary thing later. Is that okay? Fascinating. I'm certainly a "system infrastructure" person. Never did much GUI work except to excercise my back end. That's VERY encouraging. I'll explain why below. You have recommended this again. So I must be missing something. Can you please explain the reason why you use compilation to do these #if statements? Do you do that for performance reasons or what? Wow. Now there's an idea. I see more clearly now your vision of truly catering to professionals. TSGannGalt, please give me your opinion of the following: I've been a little embarrassed about the GUI I described in a previous post. It's very simplistic. I'm hearing you say, and someone else mentioned that doesn't matter. It's the infrastructure that matters most to professionals. Is that true? Wow. Please re-iterate that because I struggle (obviously) but I very much want to believe the GUI doesn't matter because I don't care about it doing anything more than absolutely necessary. NOW, TSGannGalt, you have a lot of insight and good vision. Please share you idea of how this can eventually (not right away) generate revenue. I ask because that's not clear to me in the roadmap you recommend. And nobody but nobody should kid themselves that an open source project will ever succeed even for the programmers if there aren't at least a handful of full time developers/support people who can manage the code base, fix bugs, and work on any major improvements the community needs. You used Eclipse as an example. It would have died in the water years ago if IBM didn't fund full time developers on it and get paying customer fors the commercial version of the tools. I will venture a guess. Let's guess that Quants and other professionals (much like software developers in the case of Eclipse) will often get their companies to pay for a commercial version of the system with support, SLA, and other services to fund developers. So do you think the same model will work? Sincerely, Wayne
Shazam! you sold me entirely. Enough said. I love it. You hit all the hot points and that clearly solves the versioning issue. Hey, you say you're about to implement this. What about doing it on TickZoom directly? What's your time line? I'd like to get a Wiki setup, a bug tracker, and a forum for TickZoom which can be done in a couple weeks. Then you can try it out. I'm sure you'll love it if you like what you read so far. Then you could add db40 into it directly for the community. Please share your thoughts. Anyway, maybe it doesn't matter. Personally, I'm okay with sequential read for the moment. But would LOVE for someone to add db40 into it because we're going to outgrow sequential very quickly. It would be straight forward to add it, I think. There's a Tickarray class which handles reading and writing the ticks. It already get's two TimeStamps as a start and end time/date range and the symbol to load. There's the Tick class and it's subclass called TickDOM which adds on DOM data (4 levels) to each tick and synthetic volume. For the ES, you'll find the TickDOM class already has everything you need. NOTE: One other issue with tick data is the sheer VOLUME of data makes the files so large. So the Tick and TickDOM use several optimization techniques to reduce the amount of data drastically for storing the tick data, dom data, etc. As an aside, you mentioned interfaces. TickZoom avoids them wherever possible for performance because the compiler can't optimize out method calls when using an interface. Did you know? Method calls are VERY costly for performance when repeated 140 million times for each tick time because the compiler has to manipulate the stack and copy any parameters. As you see, the system has been very carefully refined in how it handles memory and uses all the possible compiler optimizations. Anyway, this just shows an example as to why TickZoom truly ZOOMS. Sincerely, Wayne
Well, I don't mind doing it in an open source project, but as a professional software developer I get fairly picky as to what I am working on, so I'd have to see it first before committing. That little project I am still going to do anyway, in my spare time (bear in mind I develop software for a big investment bank for a living during daytime) as a "test project" type of thing. Just to see what's what, and, say, how big is a db4o db with 10 years of ES tick data (or if I need to split it into years, months etc) and what to expect if I were to, say, ask for that particular week of that year to replay it. In fact I'll probably benchmark a little several scenarios to see if db size is an issue on performance, etc. After that I'll see what I do with it, but I was indeed going to try my luck at backtesting few strategies of mine that work well in manual mode. If TickZoom is any good then sure, why not, I'll do it in there.