There is very interesting dialog between market wizard in the book of Bruce Babcock "The Business One Irwin Guide to Trading Systems". The dialog was taken from Commodity Traders Consumer Report that assembled six prominent individuals: Robert Pardo, Jack Schwager, Bo Thunman, Steven Kille, Robert Pelletier, Thomas Hoffman. The subject of the dialog was - "Optimization". Everyone of them has strong opinion on the subject - but no agreement. We should understand that is was not pure academic discussion - all of them has his own commercial interests. Don't be naive. For example Robert Pardo - sells software, so obvious he has strong optimization supporter. On other side Jack Schwager says that there is no any evidence that optimization has predictive power more than just flipping a coin. Thomas Hoffman has scientific approach to the problem. After dialog the author of the book makes some conclusions: A. limit the number of optimizable rules B. optimize over broad ranges rather than choosing the very best values C. maximize the number of closed trades in the test D. test the same parameters over multiple markets E. test the optimized system over data not used for the optimization My conclusions and my way of optimization 1. I cannot build the system without optimization. 2. My system uses rather big number of rules. Here I get into the problem of degrees of freedom. My solution (I don't know how correct it is - the common sense is not so common) I never make optimization on 1 security. I take a set of 30-40 securities and make optimization on them. In such a way I have a few thousand closed trades. As a results, I think, I have statistical significance of optimization even with a lot of rules. 3. The found parameters I check on different sets of securities (preferable from different markets) The concise process of my optimization: 1. divide securities into sets (each set includes at least 20 securities, mixed set more ~ 60) 1.1 Large Caps (DJ) 1.2 Mid Caps (S&P) 1.3 Small Caps (Russell) 1.4 Large Caps (Local Market) 1.5 Mid Caps (Local Market) 1.6 Small Caps (Local Market) 1.7 Bonds 1.8 Futures 2. Random Set 3. Mixed Set 4. A set that I am going to trade 5. Optimization/Test on Mixed Set 6. After getting best results on Mixed Set, check other set (but not a set I am going to trade) 7. If and only if I have profit improvements on almost all set I can make conclusion that my idea / optimized parameters are not random and not curve fitting. So I apply it to my trading set to check delta improvements of profit (profit with old system / profit with new one). All this on historical data of course. 7.1 If not I try to find what's wrong. Why it works on one market but doesn't on another. It's very important and difficult step. What's went wrong - volatility, trending, unusual gaps , whatever - I try to find an explanation. 8. As a final step, very important for me to check everything on different time periods. P.S. I trade only stocks/ETFs.