I've used Matlab intensively, it has a number of limitations for really big projects, it's not evident at first, and it takes too long to explain. The biggest limitation is that there is no data-management included. But it's very useful for "small" research projects. Don't bother about using it real-time, it's not worth the trouble, even if it may look good in some small demos. On the other hand 20,000 hours is a bit excessive. IMHO people waste a lot of time trying to come up with nice GUI's or interfaces with spreadsheets. Take t****station, looks good too, but it's completely useless to me. Speaking of holy grail machines, it seems that pair trading, relative value strategies, etc. for equities are getting arbitraged away... see below... any opinions? when don bright started talking about daytraders doing pairs, i saw the writing on the wall...
Not to be too disrespectful, but you have no idea what you are talking about. You could not link in 1 data feed in 6 months, let alone the rest of the problem. We have a graveyard out back for programmers that have worked here and told us 2 weeks and were still working on a subset of the problem 2 years later. What he described is not a hacked together implementation of other software packages; there is no edge there. Writing a specific system is the most minor part of the problem; we can implement a new idea in a matter of minutes to a couple of hours; the data feeds, data bases, execution links is where the work and benefit is.
20,000 hours is actually an understatement; we have 5 programmers that have been doing that for 3 years; that is conservatively 30,000 man hours, not counting the programmers that have come and gone. No fancy GUI, no real charting, no fluff; just fast, accurate and it works. I am not talking about software that can only do one thing, in one market. There are over 200k instruments out there, with each type a different or multiple database structure; with lousy documentation. Spend a couple months writing to that and then find out, "Oh, we don't use that database anymore." Now, compound that with multiple vendors and try to pick the fastest quote. Then figure out how to clean the data; after you hit a 100 lot on a 2 day old stale quote. Very complicated and very expensive; but it does work when you do get there.
metooxx: not all so-called "systems" are trivial. the data part and the execution are difficult problems, but any decent programmer can solve them. the problems are basically idiosyncracies in hardware/data provider/broker interface and lack of documentation, not so much because it's an inherently difficult programming problem. lately it's been easier than say 5 years ago though. however, some people are using very profitable systems that have taken a lot of time to figure out. of course, there are obvious "arbitrage" and "pattern" opportunities here and there, no need to waste your time if they make enough money for you and work long enough so you can keep your job.
I am not saying they are trivial, I am saying that once you have the rest worked out you can quickly adapt to a new idea, that is the only reason to develop a system like this. Using Tradestation, which does not work, as an example; lets say they have 100K programming hours into it, and you write a system in 40 hours that makes you millions. Your 40 hours is insignificant in relation to their 100K if you could not have made it work without them. And yes any decent programmer can solve them; it is just identifying a group of decent programmers, the hard part; and committing the time, resources and capital to complete the project.
that's right, a lot of time is spent in idiosyncracies. i guess 30,000 hours is an accurate assessment if you have to figure out all those things and had to trash several versions and start again from scratch. as a fun story, there is an open source C++ application out there (i won't mention its name here) you know what? it's a guy who has spent like 2 years building analysis/ data management/ execution capabilites etc. i recently look into the code and found out that it wouldn't be capable of doing certain things i am doing. not to mention there are still plenty of bugs and data type/memory management is awfully inefficient. no multithreading capabilities. if i were him i would stop trying to "fix it" and start rewriting from scratch, but like in trading, some people refuse to recognize they are wrong, cut your losses short and move on. too much ego.
We started in this business with a couple of independent programmers telling me it would take 2 or 3 weeks, at most a month to complete. The first working version, a DOS scanner was not in production for 6 months. We gave about 5 independent programmers the job, they did not know that other people were doing identical work. One guy got it finished. That is the reality of the problem. Today, our programmers could whip that out from scratch in a week or two. Experience counts.
If this were its only problem, then it is no big deal to get a gc (garbage collector) for C/C++. If memory allocation was inneficient, it would again be trivial to get a high speed allocator "drop in" replacement for malloc/new. There are even product that do both at once. This is a bigger issue, and an extremly difficult one to get correct. For example, if I were building this system, I would write so that it was not only multi-threaded, but distributable as well. Yes - this shows your real understanding of programming technique. But IMHO, shows a lack of experience in writting complex systems. In almost every software engineering project that I was involved in, we often wrote a system that we knew we were throwing away to learn what the difficulties were. In addition, something that took great pains to do in one language, became "obsolete" as other languages, or tools, became available. The best engineers understand that all programs are special purpose languages for the domain they are "modeling." First they learn the "language" of the domain by building a prototype, then they build the special purpose language that would have made the prototype in a 1/25 the number of lines of code. The special purpose language would be designed to also be reflective (in the way that java or C# are) so that anything not handled by the language would be easily implemented (btw, this is why systems like TradeStation suck - they are not reflective.) The ability to adapt to new instruments/datafeeds, etc is also critical, as a six month advantage in coming up to speed to trade a brand new instrument can mean millions or tens of millions of dollars in profit. If you just wrote something that dealt with the domain as it exists now, this would take months to adapt. If you wrote a specialized language that you then used to write the program in, it would take a fraction of the time. The best programmers are the language designer/writers. Given todays languages (without the luxury or skill to implement the special purpose language) there is no question in my mind that the best compromise in to write these sorts of tools a hybrid environment of programming languages. For example, I would use assembler of the most time critical parts. I would use C++ for the low level stuff that could not be done in a higher level language, but did not need the absolute speed of assembler, but could benefit from the portabilty of C++. Finally, for all non time critical and high level stuff, Java, (or C# as the compiler as it matures and if it only needed to run on Windows) would be my choice. The poster above that suggested Matlab as the orinating language is not differentiating between the "Model" and the "View." Systems like Matlab are great for writing the view, but are not expressive enough to write the model (hence his suggestion that you interface with a C/C++ program.) It is imperative that the "simulartor" be written in such a way as to be able to accept multiple views, or my prefered term, metaphors. I know that there are many people that are very comfortable with Excel - why alienate them? Finally, to the orignal poster - I would use your system if you designed it in one way - make it reflective. That alienates languages like C/++ and leaves Java (and possibly C#) to implement it in. If this was not an option, then make it open source so that, if I needed to implement something, I could do it with enough sweat. nitro
"when don bright started talking about daytraders doing pairs, i saw the writing on the wall... " LOL. I've been thinking the same thing. When everybody's doing it you better start looking for another strategy.
dear nitro: i myself have thrown lots of programming time because of poor design. now i am wiser and i spend more time in the design/prototyping stage. i sure have nothing against throwing code because you learn things on the way, but sometimes people have to throw code because of bad choices in design/poor planning. That's an avoidable waste of time. my philosophy is: do whatever works for you. Our platform is a mix of C and C++. I find our C++ framework easy to (re)use and don't need C# or java. Assembler we've found not worth the trouble, save for small hacks here and there and for doing naughty things . i don't know if my framework is "complex" enough for your taste. but it's running already, it's easy to maintain/update/expand/reuse, runs in windows 2000 and solaris, i don't worry a lot about programming issues anymore... IMHO this shows a lack of experience in numerical/scientific and financial programming. malloc/new are low level issues. i am more concerned about the high level stuff, how are you going to represent financial instruments, how much memory for each, how much data you keep in memory and on hard disk etc. etc. not trivial when talking about thousands of instruments with thousands of tick by tick prices/quotes and years of historical data.