I've often asked myself this question about over optimizing my systems and have avoided trying to find optimal parameters, probably to my system's detriment. I am reading a book, The Encyclopedia of Trading Strategies by Jeffrey Katz and Donna McCormick, which provides an excellent treatment of this subject and offers and answer to this nagging question. (Any mathematicians or those better versed in statistics, please feel free to chime in or correct any misstatements that I might make. I am basically talking through something I've just learned...) By performing a t-test on the resulting set of trading Profit and Losses, you can tell what is the likelihood that you have a workable system or just one that you happened upon by chance....or have over-fitted with optimization. I will use two different real world examples to demonstrate this in Metastock: Example 1: I will run two tests over the same data set, a continuous contract of the S&P 500 futures, daily bars, from April 1982 to present. I'll start with the built in Equis: Negative Volume Index w/ Opt system in the System Tester. This system runs a total of 20 optimization steps varying the Moving Average from 10 to 200 and shows a profitable "Points Only" result of 827 points (incl slippage & commissions) over the period versus a buy and hold profit of 747 points. Looks good!!! In the Results.... dialogue, I will right click and copy all of my individual trades for the in-sample test period, in this case 136 trades. These I will paste into Excel, then sort and delete the "Out" trading positions, just keeping the Longs and Shorts. I'll continue my analysis in Excel by finding the mean value of the set of Profit and Loss data generated by my trades, cell formula: =AVERAGE(X1:X137) for example. I'll also calculate the Standard Deviation, e.g. cell formula: =STDEV(X1:X137). Also figure the total number of results that you have, e.g. cell formula =COUNT(X1:X137). Now the t-value is just the MEAN divided by the Standard Deviation divided by the square root of the number of results, e.g. (MEAN/(STDEV/(SQRT(COUNT)))). Higher t-values are better. Using the function =TDIST(t-value,COUNT,1) will give you a probability. This t-test is testing for a statistical difference between two means, in this case, the mean of the P/L and zero, or no profits. Now, I end up with a probability of 10%, so what??? Well, this tells me that there is about a 10% chance that my results or profits were random, or just profitable by chance or luck. The system also has about a nine in ten chance of continuing to work in the out of sample period or in actual trading. But wait, there's more.... Optimization: If I were to take the extra step and consider the optimization on the system, then the results get worse. This optimization consisted of 20 steps, so I'd take (1-.10)=90% chance of success in the future or out of sample data raised to the power of 20, (.90^20)=11% !!! So much for a good chance of profits in the future! This is a great example of overoptimization. Example 2: I'll compare the above system to the Equis: Moving Average Crossovers w/ Opt. Over the same data with the same commission and slippage allowances, the net Points Only profit is 1109 versus Buy and Hold of 747 points. Total trades were 291 for this system. However, this system test used 60 optimization steps to arrive at this improved result...3 times as many as Example 1. The results in this analysis yield a t-statistic of 2.64. The probability of success as calculated above is 99.5%. After adjusting this for the 60 optimization steps, the lowered probability of success or profits is 92%. So even though this system has gone through greater optimization, it looks like it has a greater probability of holding up in the future. Of course, this is just one of several tools you might consider in building that winning system.