I was wondering how people determine the optimal values when backtesting 4 or more variables. For 2 variables, I thought that the 3D map is a great way to find optimal values. 4 or more variables get kind of tricky though. I always had problems picturing the 4th or 5th dimension, let alone modeling it on the computer . Im trying to identify a set of values that has a nice range of profitable results for each variable to help reduce curve fitting. Thanks for the help guys!
There are a bunch of multi-variable techniques .... conjugate gradient (often my favorite if applicable), variable metric (newton, quasi newton), simulated annealing methods etc . There are others - and several that are much better in general circumstances) but most of the time you construct your own variation of these based upon the particulars of the problem. Depending upon your goals the entire excercise miight actually be equivalent to a muli-objective programming problem. All of this can get a bit tricky and engineering the method appropriate for a particular problem can be a subtle problem..... There are many good books on optimization problems: any good college numerical analysis book will discuss these .....
The problem with classical optimisation algorithms (variable metric, Levenberg-Marquardt, quasi-Newton, conjugate gradient, etc.) is that they are based upon two assumptions: (1) The objective function is continuous (2) The objective function is unimodal Sadly, these are not true in the context of trading system backtests. That's why people resort to visual (i.e. subjective human judgement) evaluation of data plots: the algorithmic methods are inappropriate. You can generalize the visual methods to N>2 dimensions if you consider them to be "neighborhood" methods. Find a neighborhood which has all points of the neighborhood receiving a "good" backtest result, and no other neighborhood has a better overall score. What is a "neighborhood"? You could decide it is a circle, or a rectangle (in 2D). The generalization to N dimensions is a hypersphere or a hypercube. What is an "overall score" for a neighborhood? You could decide it's the mean of the scores of the points within the neighborhood. Or you could pick the median. Or (here's a mind bender) you could pick the WORST score in the neighborhood - - - now you're doing a minimax optimization. Oooh, aaah.
Actually, your blanket statement is not really true .... you can use some of these techniques in some situations depending upon your goals and needs. They can be statistical or not, or can be combined with statistical methods: It really depends upon your understanding the problem. If you dont understand the problem then really all bets are off and none of the techniques you mention will work .... So ... its best not to really wander down the halls of "backtesting" - which itself is a term largely pormoted by companies selling "backtesting" software features or systems - unless you understand the goals and mechanics of your data analysis process, and the real problem that you are trying to solve. .....
unless you want to write optimization code yourself, you may need optimization software - you can search on ET, there have been some threads on this, all the best.
But isn't backtesting always necessary to verify your system? Granted, you should understand what is going on, but that's a given for applying any strategy, right? AFAIK the only post-optimization method available by most "advanced" trading software is the ol' brute-force trial-and-error: run through every user-specified combination of parameter values and list them in order of equity returns, the only optimization function provided. At least that's the way MetaStock worked when I used it. Only now that I use a spreadsheet has actual optimization methodology become a factor. Usually when I get an error message, I just hit a different method option button at random. It often works after that.
You might want to consider some neural network software. Their attraction is that they can pull non-linear as well as linear relationships out of your data without requiring you to be up to your waist in algorithms. The software makes it fairly easy to create, train and test a multi-layer backpropagation network which is typically used on financial time series. I played around with some software from NeuroDimension that let you create the network and then compile it as a DLL. It could then be called and used realtime by other apps. Good luck.
You can do a search here in ET and find comments from former NN-users expressing the failure of NNs to add to their bottom line.
kut2k2 - I have only played around with them but have to agree. In my experience, they fared better as filters rather than triggers. In their defense however, I have known people in the past who claim to have used them successfully in automated trading.