sounds reasonable given the size of the trading instrument universe and the possible way each of the instruments can behave at any time...
the trouble is the population size is extremely large - if you take as a population the size of all possible security setups the trading instruments can go through....
does this mean that if want to test your newly-found edge you should concentrate on the recent prices? That sounds contrary to some of the opinions in the trading books that you should test your system over a long time back...
As mentioned... testing a parametric system and a edge-based system is tested differently. Everyone has their own set of definition of what an edge is... I wouldn't consider what most people consider an edge (basically, a tested system using statistics that had a positive result). An edge for me is an exploitation of the market structure tendency. For example: SOES Bandits. 99 looters. Inter-broker like FOREX pricing arbs. Millenium Arb. and etc. etc.
Depends not only on the sample size, but also on the type of strategy. An example would be selling naked OTM puts - you could be profitable over 1000 trades, then blow up spectacularly with one big loser.
1. I would not wait more than a year to test any system. Life is too short. 2. I would wait the year because it runs the system through one whole fiscal cycle, including a good sampling of unexpected events and ranges of volatility and volume. 3. I would not test, let alone trade, any system that holds overnight. I don't gamble.
These are executional edges. The list of names are parts of the market structure in which they were found.
If your system only gives 10 signals a year. Test it going back 10 years across 10 markets. Or 20 years with 5 markets. The more the better. But 1000 is the minimum, ideally you want several thousand instances in your back test.
Should your system work in previous years (based on the fundamental idea driving the system). If yes, and if you have enough sample trades from past data then test over as many years as needed. Otherwise look into developing synthetic data.