It sounds like your results are going to rise and fall based on volatility. The stocks that are giving you the strongest results are most likely the most volatile. I would not toss anything out of your results, but look into it to see if this is the case, and when it is happening. (Possibly your system would benefit from putting ONLY the most volatile stocks on your watchlist. Who knows?) When dealing with a portfolio of stocks, the biggest value in backtesting isn't to bash the data so much that you end up with results that are not realistic, or can't be traded in the reality of the day-to-day market. Rather, it's to get a good understanding of what's happening, and in what market environments, and then know how and when to make adjustments. Because the market will always change, and your system will need to change with it, or die. Although you only have data for a year, the good news is that during the year we've had extended periods of very low, and very high volatility. It's a great test bench for any equities system. So it should not be difficult for you to run your backtests during each of those periods to get a sense of relative performance, and what to expect in the future. Based on the results, you may want to introduce an indicator, such as ATR on the S&P, that measures overall market volatility and adjusts the parameters of your system accordingly.
I am still pretty new at this but wanted to avoid over optimization. In theory the system should work the same regardless of which stock is being traded so I optimized with just one stock (which actually had data from December 2006 to December 2007 because of when I started) then tested on three and re-optimized (this required only two small tweaks). I have now taken that system which was optimized over three stocks over one year and tested it over 40 additional stocks (I will do the entire S&P shortly I just wanted to test it over a smaller sample to see if I needed to continue changing the variables, which I did not do). Those 40 stocks which I picked are 4 from each sector, and within each sector two that ended the year up and two that ended the year down. While this data is fairly similar to the original data that I optimized on because it runs from Feb 2007 to Feb 2008, I am hoping that the 40 additional stocks would qualify as "out-of-sample" data even though they are 83% from the same time period. I will be searching more to examine whether this assumption is correct however. I am still working on getting data off of Open Tick to use for backtesting so soon I should have a better idea of how accurate my backtesting has been. Also testing the entire S&P should help. I am a little nervous since my average gain over these 40 is 74% but the standard deviation is 39%. My numbers also appear to either not be a standard bell curve or else 40 stocks is just too small of a sample because with this mean gain and standard deviation, 95% of my profits should fall between -39k and 1.5M, but my lowest profit is 230k. In addition this is with taking out any trades that account for over 8% of the profit of the system. The problem with this is that there is still a carry-forward compounding as if the trade did occur. To attempt to account for this I actually subtracted 130% of the trades over 8%, not very scientific I know. I am hoping that I can narrow down the stocks I will trade in order to reduce this standard deviation. Using volatility is a pretty good idea, I will be looking into that. Thanks again for the feedback.
Here is a book that will answer a lot of your optimization questions: http://www.amazon.com/Trading-Syste...=sr_1_1?ie=UTF8&s=books&qid=1206059758&sr=1-1
^^^I will look into that book. I have Pardo's but have not had the time to finish it yet as my finals just ended yesterday. I guess I probably should have finished it before I came on here, but I have gotten some good advice.