I was a BioComp user for over a decade but abandoned it some years ago for serious trading. It's probably no better or worse than other tools like NeuroShell that are marketed to traders who have little in the way of data mining skills or training, which isnât saying much. A few years ago I challenged the CEO of BioComp to a "Pepsi Challenge" in terms of trading results. The challenge was take a standard price data set with the same dependant and independent variables and run it through BioComp and see what the results are in terms of trading performance. The same set would be run through NeuroShell and similar competitive systems. Carl declined saying he was much to busy to allow or support a comparative evaluation of his product. I see he now has some time to post here. What about it Carl, how about a standard objective evaluation of your products performance? I'd be happy to supply the data set and we can get some of the old participants from the Yahoo Neural Network group to act as judges. For anyone wanting my suggestion of software applications: forget the stuff promoted to traders. If the folks who create it and market it could make money trading it, they would. They can't so the sell it, which is a major undertaking with customer support, marketing plans, ad buys, time posting to newsgroups under various identities, etc. Instead get the best general purpose data mining application you can afford. That is the kind of systems used by major corporations for credit card fraud detection, oil exploration and similar mission critical applications. This stuff actually works on complex problems. Jerry030
Jerry: Four posts in June 2006 don't indicate that Carl has time on his hands. The two equities/futures-oriented products Carl offers, Profit and Dakota, are not Carl's main business. I'd suggest you go to the BioCompSystems web site to get up to speed. I doubt that Carl is a regular reader of ET. James' posts here are an indication of how Dakota works. Jack
Jack, Funny Carl said the exact same thing to me by phone "two equities/futures-oriented products Carl offers, Profit and Dakota, are not Carl's main business." as the reason he didn't want the product objectively evaluated. Are you a BioCOmp employee? Jerry
Jerry: No. I own Dakota. As you are probably aware, there is a private Biocomp forum and Carl is an active participant there. He takes a hit from time to time for delaying new features he has told us about. His reason - the award-wining process control business- was demanding a lot of his time. I do not understand why he would tell you that just because he had two products available in this market he did not want them to be objectively valuated. Something is missing from your recollection. I attended a training seminar when Dakota was being introduced. Present at the meeting was a senior level manager from a very large Wall Street firm. They are still in business! This guy was from (or headed) their equities analysis group. He was there to learn about Dakota. His firm had used and licensed multiple copies of Profit for years. Indeed, if I remember correctly, the firm has one full-time analyst/programmer whose job it is to develop models used by Profit for other analysts in their areas of specialization. A friend who purchased Dakota on my recommendation and attended a later training seminar told me the same guy and his system analyst/programmer were both in attendance. I guess the firm now uses both. Jack
I bought dakota through my employer. The swarm concept is awesome and interesting but as far as using it to trade real money - I would not trust it. In the markets, evolving/adapting systems generally don't work over a long period. I bet a lot of evolving algorithms failed in the first week of October (for stocks anyway). The way to go is to have a number of simplistic static linear rules, each rule uncorrelated with the others (or low correlated).
Jack, Next time you speak with Carl bring it up. In corporate data mining, which is where Iâm from (banking and financial services) there are all kinds of standard data sets used to compare and calibrate the effectiveness of both software applications and analytical strategies. CMU and other academic sites have these freely available. I proposed to Carl, along with other vendors, that we do a similar standard test to rate the effectiveness of their product in developing trading systems. In multiple phone calls with Carl he danced around the issue with various objections until finally settling on the one that he and his staff were just too busy to load a data set and let it run to completion. Youâd think someone with an excellent product would want this kind of "Consumer Reports" testing. They come out on top, with results that make money and have a great market endorsement....."Four stars in independent testingâ. Honda uses it to sell cars...but of course their cars actually work. So tell Carl if he's got time these days I can round up the group we had willing to work on this project. It would be fascinating to hear his response this time Jerry030
Jerry, I have been lurking this thread since the beginning, but I feel I must now post. I think our conversation went a bit differently as I recall. This is what I said to you by email about your proposal (cut and pasted from my outbox) ========================= As I think I mentioned before, that will be an interesting challenge. Just one aspect is the large human element. User A can get fantastic returns with a particular product while User B gets poor results. How it is used, what user selected options, data used and its handling, indicators used, ... Another aspect is the definition of "effectiveness". A plethora of metrics makes it difficult to come down to a product ranking, which would be the first thing people would look for. Those metrics would have to be calculated by you in a test bench, as the means of calculation vary by product a lot. Drawdowns are an interesting example. Another aspect is what "trading system" is used, which may be product independent, if a product even has a "trading system". Another is money management vs. performance. For example, some systems may compound returns while others can't. Some may have numerous order types simulated, while others just at the market. Some have certain kinds of stops, while others don't. Another element is if there is a quantified comparison of products, any judgment must be backed up not with just metrics, but evidence-based proof, demonstrating that it was not just luck that Product A did better than Product B on 1 or more securities during a selected time period. If you haven't already, I would suggest reading David Aronson's book, "Evidence-Based Technical Analysis". Excellent. We may give it away with new Dakota purchases. A "must read" in my mind for all traders and is very applicable in this case. Another thing is the capabilities of one product vs. another and whether the products can legitimately be compared. Another thing is the target securities "class". If you test on stocks, some products will do better than others. If you test on futures, a different rank. If you do forex, yet another. So you may need to segment the test suites. I could go on. However, if you do this well, and if it is actually and humanly possible, it could be an excellent benchmark. ========================= Thanks. Carl
James, Thank you for having the courage to publicly "wind surf during a hurricane" and to stick with it when the storm surges take you under water. Glad to see you are above water again. Thanks, Carl
Carl, In the end your final response was that you and you staff lacked the time to load a standard benchmark dataset and run it. Is your time still scarce or are you interested in some objective comparisons that traders may find of value in making software purchase decisions? Best Regards, Jerry
I think you misunderstood. That was not the issue, nor my response. My point, as I sent you in the email I posted above, is that there is no such thing as a standard benchmark and I doubt you could come up with one that would give any meaningful comparison. Since Dakota and Profit are software tools, not trading systems, I would like to say politely that it's a non-sequitur. As an example that will be clear to most everyone who trades, go ahead and run a "standard benchmark dataset" in Tradestation. You can't. You have to set up a particular strategy, trading system, stops, settings, equity objectives and metrics, etc as defined by the user. Now, if the software was a canned trading system, then maybe it would make sense, but even in that case I think coming up with a test suite that runs in a work bench that you would have to create to have equal calculations and metrics, that gives a valid comparison of canned systems, would be very difficult, because you have to prove one system is better than another. Our products are not canned trading systems, nor automatically create trading systems, so I humbly suggest what you are asking for, while great in intent, is not anymore doable than you could with Metastock, Tradestation or other tools. Thanks, Carl