The pressure has been strong from PM's and private email to make the engine run in C++ for a variety of reasons. In the last 24 hours, I was digging into how to do it. So as to make sure that's an option. Upon closer inspection it has become clear that C++ will likely be significantly faster. One reason is the "reference counting" that the garbage collector must do "atomically" which means thread locking. In C++ you can control when that's needed (and it's not needed in most of the engine). So it's possible that will speed it up significantly. It turns out there's an automated tool that converts C# to native C++. It's lacking one important feature after paying for it but the author said he'll add it within 2 days. That will only leave me to write the "glue" code from C# to C++. (It will also be easy to make Java glue code at that point too!) So this means that the following will stay in C#: (The rest of these components could rather easily be redone using Java or C++. Charting especially needs it because that's very Windows specific right now. In Java you probably want to use JFreeChart as the foundation. Is there a c++ equivalent?) The following are outside the engine and staying in C#: Charting. Optimization algorithm. Performance Stats. Black Box Server. Execution Server. Indicator Library (very few). Function Library (very few). Custom Trading Rules. What will be cool about this is a much greater interoperability. Eventually someone could develop strategies on windows using a the C# GUI but write trading rules in Java or C++. And then roll those trading rules to the black box without any code changes since the engine will run on any server using the GNU C++ compiler. Any thoughts? As far as release date, I'm keeping that posted on the home page of tickzoom.org. Currently January 11th at midnight. But this could easily bump it out another week if I run into any glitches. And it depends on the convert tool company adding that new feature to convert C# properties to C++ getters and setters. It short I decided it will be better to knock this out before releasing because once released, I 'll be very busy with answering questions. Wayne
Several questions: What's your expectation about the performance gains - will it be worth the effort of porting and re-testing the whole thing ? I assume the biggest performance bottleneck is a backtesting part - are there many objects created/disposed during the run ? (in this case there are may be other solutions - algorithmic ones - to address this, instead of relaying on another language). Is the "performance" thing is the only reason you are going to rewrite your engine or there are others ( like easier integration with another C++ base libraries like QuantLib or Tree )? Would you be doing this for commercial software project which is just about to be released ( like alpha stage ) ?
Yes it will be worth the gains. Let me explain why. About re-testing. What most excites me is that this is the first software system I built for myself (not business) where I fully committed to writing an automated test suite from the beginning and keeping it up-to-date. So there are two levels of tests. 1. Integration tests. These won't change because they run at a high level and will still be in C#. So they're an excellent automated and instantaneous regression test. 2. I have lower level unit tests which I will have to convert to C++. However, even after converting that will save tremendous time over writing tests from scratch or, worse yet, manually testing. So in short, I'm excited by how quickly TickzOOM will be able to implement new and major features quickly and reliably. I think that's going to be the most powerful feature even though it's behind the scenes. In fact, without the auto tests, TickzOOM would never be what it is today. and thanks for an automated tool for porting code by tangiblesoftwaresoluti ons.com it should go very quickly especially because the engine doesn't use much Windows stuff, it's just data structures and home grown code. NEVER. Not a single object gets created or disposed during high speed. Garbage collection is never used during high speed code. Still I could disable reference counting in C# also by using pointers more than GC handles for objects. But in C++ the converter automatically makes everything a pointer. Correct. I already implemented the algorithms. C++ will only at most be 10% faster (and it might not be faster at all but we'll see. So far the only advantage I see is the reference counting thread locks that can be eliminated, and that's minimal.) One thing is for sure. It would be impossible to get this performance from Java since it disallows pointers (right now TickZOOM uses C# pointers in unsafe mode) and Java doesn't have a sizeof() operator, etc. C# in reality can be used as a low level language like C++. And so the language change itself won't speed it up much. The main reasons are portability and easier, more natural integration with C++ and Java as well as C#. This is FREE software. It's also "vaporware" in a sense since I haven't released it yet. Thank you for your question. I enjoy thinking these things through. Sincerely, Wayne
If it's of any help I have in the past benchmarked very simple algorithms in both C++ and C# and found the C# versions to have a speed penalty of around 30%. This figure may not be relevant here as that code was doing very few memory allocations, it was mostly integer arithmetic and boolean tests.
Perhaps you meant "One thing is for sure. It would be impossible FOR ME to get this performance from Java..." ? That aside, my compliments on having the enthusiasm and skill to create what sounds to be a decent bit of software.
I don't know Wayne...personally I think you should just release what you have and see what happens. I notice with alot of open source projects, a good idea will get going but then since there is no limits to what you can do and no time constraints, alot of great ideas are left to rot on sourceforge by trying to create the "ultimate masterpiece" instead of small incremental improvements to flawed but usefull software.
I feel the same way. My goal with TickZOOM is to make it free to the masses but eventually (like many open source projects) have businesses fund the project. That's the only way to have a serious support team with phone support and technical assistance. However, some business owners in this industry have kindly advise me that will never work. They explain that most businesses in the less regulated financial industry will just take the source anonymously and never fund anything. So the point of that advice was to only release binaries and allow people to get the source code upon request and with proper identification and signing of paperwork to make it more enforceable than anonymity. In the end, my choice was to keep everything available via anonymous download except the CPU intensive part of the engine. Furthermore, rather than try to "obfuscate or encrypt" the C# assembly for the engine to hide the source which will hurt performance, I decided to just convert it to C++ which will, if anything, improve performance or keep it the same. The C++ source code will still be available too but with strict controls over who gets it. So you see, this solution will work for most users because everything people usually care about like the file formats for tick data...execution server, black box server, charting, optimization algorithms, etc. etc. are all open source and available for anonymous down load. In fact, most people won't need or even ever want to bother with touching the c++ source for the engine. So in the end, this has the benefit that makes it possible to use an ordinary GPLv3 license for all the rest of the code which will better facilitate sharing. Now for the downsides... I sincerely feel apologetic to those, who like myself really love anonymous download open source. But reality has to kick in at some point. And what is most aggravating to me is this delay to releasing. And I'm seeing that my 10 year old expertise in C++ isn't helping me much. Plus gluing C++ to C# is no walk in the park! I wish the world was a nicer place where everyone was honest and fair. It would make things so much easier. I hope many people see this as a viable compromise. Several via PM and private email say this works for them. So we'll see. Sincerely, Wayne
Back to the topic of this thread... I just found a tool called SWIG which automatically generates "glue" code from C++ to many different languages include C#, VB, etc via COM and Java too. So that might turn out much easier than expected. I still have hopes to either finish or get it mostly working by this weekend. The conversion tool company finished fixing that issue for properties to setters and getters yesterday. So things are moving forward! Sincerely, Wayne
What i don't like is that your project still takes a different direction every week. If this continues, how could i invest work either for personal purposes or contributing something if i don't know whats the situation a week/month from now when your next idea comes up that turns things around again? You need a plan, and stick to it in my opinion. It just doesn't instill much confidence if the project-leader changes language a few days before wanting to going public. With a tool that doesn't exist yet. To a language you didn't use since 10 years. I don't want to come across as negative, i still think the basic idea is very good, but i am a bit skeptical about the endless zig-zag. Hopefully i will be surprised in the end tho. Regrads, P.