1. In addition to the Grail Genetic Optimizer and Grail Walk Forward Optimizer that TS just bought and will incorporate in TS9, what other products exist to automate the optimization, walk forward, in-sample, out-of-sample work? 2. Rina Systems has a product called 3D SmartView that lets you look at the results of an optimization run in a 3D graph for $500. Wonder if there are other competing products? (Amibroker includes this feature in their platform for no charge!) Brooks
I actually prefer contour plots (rather than 3D diagrams) to view optimization results. Gnuplot (free software) can make them; so can Microsoft Excel. Trading Blox, another trading system testing package, offers both. Here's the same optimization run presented as a 3D diagram and as a contour plot; I prefer the one on the bottom.
Thanks for the graphs, MGJ. Actually having the option of both is nice. The 3D shows spikes well. The contour eliminates some of the tilting necessary to see all the data. I'm also curious about the scale and what's meaning measured. Brooks
These are just two different ways of visually presenting the exact same set of underlying data. The data itself comes from an optimization run over "n" different parameter combinations. The data is conceptually a three column table Code: Optimization | Optimization | Trading System Variable #1 | Variable #2 | Test Result ==============|================|================ x1 y1 GoodnessMetric 1 x2 y2 GoodnessMetric 2 x3 y3 GoodnessMetric 3 . . . . . xn yn GoodnessMetric n The plots do nothing more than display (Variable 1) versus (Variable 2) versus (Goodness). The idea is a simple one. In the example plots above, (Optimization Variable 1) is the first number of days in a MACD indicator, and (Optimization Variable 2) is the second number of days in an MACD indicator. Variable 1 runs from 50 days to 100 days, and Variable 2 runs from 200 days to 400 days. In the example plots, the Test Result being plotted (the "GoodnessMetric") is MAR Ratio. MAR is an acronym for Managed Account Reports, one of the earliest CTA performance-tracking newsletters, from the 1970's. For the purpose of ranking the managers they tracked, the editors defined the "MAR Ratio" to be (CAGR% since inception) / (MaxDD% since inception). This ratio is still used today by allocators and fund-of-funds managers and the reporting websites that present track records. Lots of trading system testing programs present the MAR Ratio as one of the available output statistics. In this particular test with this particular system trading this particular portfolio over this particular slice of history, the MAR Ratio varied between +0.12 (dark blue) and +0.36 (bright red). The colors are chosen to resemble temperatures, with blue ("coldest") representing the lowest values of MAR Ratio, and red ("hottest") representing the highest values of MAR Ratio. These plots were automatically created by Trading Blox. However, once you've exported the test results into a three column table like the one shown here, you can feed that table into lots of different software packages for plotting. Excel, Gnuplot, and PSIplot will all do contour plots and 3d plots and much more. You have other options besides plotting-from-within-the-trading-system-testing application. If you wish, you can do the optimization run in Tradestation, make a 3-column table of results, feed that table to Excel, and have Excel make a contour plot.
Here is one that I looked at, but by then I had already purchased grail walk forward opt. Greg is very active in TS fourms, you may want to look him up. Will only work with intraday data. http://www.stratopt.com/index.htm
I took a look at the website. The developer claims that "A strategy that does make it through a walk forward test with acceptable results may still end up not working well in real time testing for one reason or another, but one that fails the walk forward test in the initial runs will fail in real time with near certainty." I agree about the first part of the statement because it is empirically verified. However, there is no support whatsoever for the second part of the statement "...but one that fails the walk forward test in the initial runs will fail in real time with near certainty." This argument has been falsified many times. Many of us have seen strategies that failed in forward testing but then wished to have traded them real-time. Forward testing is nothing more than regular testing backtesting+forward testing + optimization + anything you like = just historical testing no guarantees either way.
I think I understand what you are saying. Not sure I agree 100%. What are thoughts about the following tool, in particular what do you think about the idea of stress testing parameters? Thanks. http://www.thegrailsystem.com/grail-trading-system-faq.htm from the author of the grail wfo: Even though the genetic/walk-forward optimization approach is not as prone to over-optimization risks as other genetic optimizers, the GGO also introduces an additional measure to promote stable parameter selection during the initial genetic optimization stage. This is achieved by performing a stress test on parameter combinations to see how performance would have deteriorated if all the parameters were changed a certain percentage.
In the book Market Beaters by Art Collins, Murray Ruggiero is quoted as saying: "If you don't like the neighboring numbers, you've got a problem, because odds are, you WILL wind up with the results of the neighboring set of parameters." Stress testing sounds like a very good idea.