There's a MA Crossover thread in the same forum, so this one looks at the problem from a different angle Assume you've build an algorithm (e.g. genetic) to derive a MA crossover strategy that works well for USD/EUR USD/JPY and GOLD but the strategy works terrible for USD/CAD OIL and SPY Assume you've over 200 trades in each market/pair with this strategy and the system was "optimized" for overall profit (completely ignoring drawdowns, etc..). What do you do if you feel comfortable with the system (and the drawdowns)? Do you just trade the markets where the strategy works well, or is the result just a highly curve fitted strategy which has no significance whatsoever?
I test nonoptimum parameter sets to learn if the system shows profitable results under varying conditions. I test my system by adding noise to price values and observing trading simulation results. If I need to be feel comfortable then I do not trade.
You could ask the question "If I had used system development method X back on 12/31/2007, how happy would that suite of systems have made me in 2008 and 2009?" The simple way to answer the question is to actually run your system development method X but only for historical price data up through 12/31/2007. Create your optimal suite of systems with your optimal parameters, using your method X, on historical price data up to 12/31/2007. Then test it on price data after 12/31/2007. Look at the out-of-sample test results; do they make you happy? If so then it boosts your confidence in system development method X. If not, then it reduces your confidence in system development method X.