I heard about "Walk Forward Analysis", to me it's just like a dynamic rolling-window out-of-sample test coupled with optimization. The result of this "Walk Forward Analysis" is a set of "optimized" parameters that are suitable for all history(it's robust, but kind of conservative). Am I right? Any good ways of avoiding over-fitting or over-optimization? Thanks!
The first caveat is don't optimize if the bare strategy without stops isn't significantly profitable.
create a complexity number for each function... give the functions with the least complexity the most risk capital, and test the results. I have no idea what the results you'll get, but by evaluating the complexity of parameters should avoid over optimization You might be able to evaluate different strategies at the same time as well: http://en.wikipedia.org/wiki/Cyclomatic_complexity
One way is by using price action formations. You replace the problem though with selection bias. One can have either curve-fitted signals or signals that are selected over others but you cannot avoid both, at least I do not know of a way of doing that. I prefer selection bias over curve-fitting. Examples (1) Curve-fitting: SMA(a) > SMA(b), adjust a and b for profitability (2) Selection-bias: close > high(2) has been profitable but not close > high(1), etc. You select the profitable one.
When optimizing system variables, I look for "fragile" settings. This is where say 37 is profitable but 36 and 38 perform abysmally. The 37 setting is probably profitable due to one or two lucky trades it took advantage of. I like to see ranges of system variable values that are profitable. Then I pick one in the middle rather than the one that produced the best results.
Agreed, which is why I prefer thinking of this process as evaluating robustness rather than optimizing parameters.
There is no such a thing as a "somewhat pregnant" woman. The same goes about optimization. The issue is whether a system is optimized or not. Any system that utilizes a function to generate signals is optimized with respect to some objective function.
Well, right. Even a moving average crossing system has to be run through an optimizer to find which combinations of MA periods work and which don't. You can't just pick 2 at random.
Well, the point is that regardless of the choice of paramaters, such system is always optimized. You can always find an objective function that any such system maximizes. The lesson is that if you have any parameters at all in your system, it is optimized, whether you actually optimized it or not. I tell you, very few understand what I wrote above but I know a few do.