I am trying to design a strategy by examining a great number of seemingly unrelated factors. This could be applied to trading systems, but it is not limited to them. I make (automated) tests against thousands of different factors (about 300.000 factors). Some of these factors may be: Day of week, Day of month, Above/below 30 day moving average etc. I came up with about 1000 factors, each one of them giving very descent signals. The combination of all 1000 factors gives great results. Now suppose I only have 6 months worth of data. I use 5 months worth of data to design the strategy (and come up with the 1000 factors). As I said, the results look great (on paper). Then I try to verify the strategy on the 1 month of data. Disaster! It looks like the strategy is designed to LOOSE. What steps should I take? Obviously I have fallen in the curve-fitting trap. But how do I get out of it? After reading on optimizing automated trading systems, one advice is to try to examine the big losses. Why something did not work. Well, this is not applicable in this case. One idea is to test individually each one of those factors against the 1 month data. IF they are profitable there (I see consistent results), THEN I can pick this factor as a profitable one. So I may come up with 300-600 factors. And then, their combination will be my system. Slippage and commissions are not an issue. Any ideas guys?