I test strategies by guessing, say, the optimum values for the RSI. As an example, I look at three stock's 10-year data, and I see what parameters might work for those three stocks best. Then I apply the same method to, say, a 100 stocks. If my results are good, I keep the strategy, otherwise I disregard it. How do I know if I am over-fitting? Secondly, a yearly return of 50 percent in the last ten years for the stocks in my list...is it good enough? OR, back tests are usually better?
I tested how my strategy performed in the past two years separately. I got similar results to what I was getting for the last 10 years.
Note that I am not optimizing my strategy over all my stock universe. I just look at a couple of stocks, see what feels correct, and then apply that strategy on all the stocks. I don't test and retest on all the stocks.
You can test all you want, and offcourse rigorous testing is a good thing to do. But there will always be a time when you have to test the water. What helped me in the past was just setting aside a small sum to test the water. I said goodbye to the money asuming my strategy would immediately stop working when I would use real money. I think overtesting is sometimes a waste of time. What if you can trade your method for 2 years before it stops working? You will earn more if you do it for 2 years and then stop ( assuming no black swan). Also, the time lost without real trading can cost you money in the long run. You have to learn to deal with all the emotions and your 'inner demons' involved. Just start small, learn on the way, and if your method keeps working use gradually more money, and your account will grow over time.
If RSI is your primary signal generator, look at the dates when most of your stocks are signaling an entry. If you're generating a large number of trades at the same time, it's likely that your results are just keying off significant market moves during your test period. Trading 'baskets' of stocks can be difficult in those situations, because you're left with the decision as to how to allocate capital, and to which instruments. Try running your backtests on levered indexes.
Ask yourself - Why would a strategy work for those 100 stocks and not another 100? If there is a particular reason - is it likely to persist overtime? How would it do in a prolonged declining general market? 2008/2000 like. How would it do during a flash crash like 1987/2011? Do you have sufficient # of trades? 1000+? Does your data include delisted companies? Survivorship bias. What is your MaxDD for that period? It is typical to have 1:1 return to DD ratio. If that's what you have - expect to see 2x DD live at some point. So by the time you hit it you might blow up your account, which means you are sizing too aggressively. How much capital is used to generate those returns? You backtest might include unrealistic leverage Val
I can't remember if I read it or heard it in a podcast but I have found myself repeating this phrase in my head many times; " if you are wondering if you have ventured into the realm of curve fitting, then you are already there" (apologies to whoever said/wrote this for butchering it). Practically, I also use another approach (also not original) of only either doubling or halving the values of an indicator (eg try EMA(C,100)....then may try 50 or 200...if no joy then move on to another indicator/idea). I assume all my strategies are curve fitted until proven otherwise. For stocks, I use a different trading universe (held on a separate laptop to stop me "peeking" during strategy design). I guess the other key (for me) to tell if I have curve fitted is if the backtest equity curve is too good. Hopefully there are a few years of poor (or dare I say it.. negative) returns in that 50% cagr ? Survivorship bias is a major trap as Val has already mentioned (along with all his other ones).
Over-fitting is unavoidable, as the markets keep changing. How to minimize over-fitting is a better goal.