You should take his/her comments with regard to using Excel more broadly. What it implies is more important than his/her specific use. The point is: To really do some beneficial backtesting you are going to have to get your hands dirty with some data analysis and manipulation - as opposed to having a pre-packaged program accomplish all aspects of system development. "Don't optimize, because you will curve fit," is such a hackneyed and trite statement it makes me vomit. Optimae emptor....optimizer beware. Constrained optimization is very useful. Incorporating another issue raised in this thread: the use of multidimensional data representation. The use of 3D graphs can be very useful in evaluating 2 parameter systems (assume profit is on the z-axis/height). One can simply look at what looks like a topographical map for REGIONS of higher profitability, as opposed to single point that happens to be a global max that is surrounded by unprofitable regions. By assessing a bit more of the "larger picture," one can avoid the pitfall of settling on what happens to be a historical global profit maximization point surrounded by unprofitable points in the 2-d region of parameter possibilities. Look for a range of profitable parameter settings.... Let's say Trader A has a system that is profitable (backtested only). He/she has come up with specific parameters by a few random choices. Trader B has come up with a system that optimized his parameter settings such that he maximizes whatever objective function (some risk/reward representation, likely) out of the entire space.......Now, ceteris paribus.....Who is to say that Trader A has happened upon a more profitable surrounding region simply because he didn't use optimization..........in fact take a large enough population of Trader A's and one Trader A has likely settled on the SAME parameter settings as Trader B, only randomly.............ignorance is bliss I suppose....
Maybe I sounded like I'm against optimization. Not really... I believe it's a matter of how you evaluate systems. I use XL and run optimization on a basket of systems or for a portfolio(I'm really careful with this). Haven't run optimization for a parameters and most of my systems don't necessarily use parameters. I have seen more failure than success with optimized system developers. Anyways...This is an interesting subject..... Can anyone provide a picture with some explanations on how you look at the 3D graph. I would like to learn about successful application of optimization. Thanks. Trend
Tomasz Janeczko, the chief designer of AmiBroker, had some good advice I read somewhere. He suggests optimizing your system on one set of data (a certain date range, for example) and then testing it on a seperate set of data. Also, in terms of "real world" applicability, you should keep in mind that no trades a system generates are going to be of any use if you can't get them off at the specified prices. That becomes more and more of an issue as your time frame becomes smaller. A system for scalping, for example, might show you a 1000% annual return. But if you can't actually get those trades off...what good is it. sax
I don't. I backtest / optimize with Tradestation and export all the results from the optimization to Excel. You could backtest and optimize with Excel if you want to write all the VBA code, I tried it at one point, too much work, too many gotchas. Max
The 3D graphing is not absolutely necessary for evaluating a system but it really makes it a lot easier and serves to educate the viewer to the possibilities. Mathematically speaking it is totally clever. There used to be a free one but I can't find it anymore, RINA systems has one. Max
I threw out a lot of systems because it wasn't applicable. The one's I threw out were too hypothetical. Trend
Max, 1. How are sorting the data when you use XL? Can you provide the name of the free software? 2. How do you choose which parameters to employ? Do you have a specific system or it comes out of experience? Thanks. Trend
If you can do it, use "walkforward" optimization. If you do not know what it is, it optimizes on one set of data and test on another set, not used during the optimization. The results from the test set should be similar to the results of the data set used during the optimization. If any Tradestation developers are reading this, "hint hint" for a valuable addition to the software. Doug S
Well...need to get back on topic. I'm getting a lot of things about optimization but does anybody use MonteCarlo Sim? I use this a lot when I evaluate systems by random testing trades to see if I have a system that won't rely on outliers. I usually run 1000 different tests. I found this tool pretty useful. Some other things that I like to do is to use WLD and TS to evaluate systems. I convert systems on both sofware's. I like TS's analysis and WLD has some other interesting features. These 2 sofware has different features which helps. When I employ both I see the bigger picture because I get different results(slightly different) with the same system applying it on same data. Good Trade. Trend